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FROM  THE  SPONSOR 


Crosstalk  would  like  to  thank  309 

Since  the  beginning  of  early  civilization,  humanity  has 
established  goals  and  succeeded  in  completing  remarkable 
engineering  accomplishments  that  pushed  the  boundaries  of 
what  was  considered  possible.  One  needs  to  look  no  further 
than  the  Great  Pyramid  of  Giza  (2584-2561  BC),  Lighthouse 
of  Alexandria,  the  Parthenon  (447-438  BC)  or  Chichen  Itza 
(c900  AD),  for  notable  ancient  examples.  In  recent  times,  we 
have  gone  from  creating  the  monuments  of  ancient  times  to 
engineering  complex  architectures,  structures,  electronic  sys¬ 
tems  and  artificial  intelligence,  such  as  multinational  undersea 
tunnels,  artificial  archipelagos,  the  International  Space  Station, 
and  IBM’s  “Watson”  -  the  new  Jeopardy  champion. 

Major  projects  are  not  new,  but  the  way  in  which  we  now 
manage  projects  has  evolved. 

The  field  of  project  management  has  continually  transformed 
to  address  new  challenges,  primarily  an  ever-increasing  growth 
in  complexity  and  scope.  Innovation  has  led  to  new  ways  of 
managing  interrelationships  between  specialists  performing 
a  vast  number  of  different  tasks.  Hardware  and  software  are 
now  coordinated  in  concert  within  a  system  design.  Systems 
of  systems  deal  with  our  increasing  need  for  instantaneous 
information.  Compartmentalization  of  core  best  practices  for 
project  management  activities  include  integrated  planning, 
organizing,  resourcing,  directing,  monitoring,  and  issues  resolu¬ 
tion  allow  for  greater  reach  and  control  over  simultaneous  and 
multifunctional  tasks. 

Software  system  development  has  brought  its  own  unique 
challenges  to  the  table,  such  as  its  intangible  nature,  clear 
requirements,  feature  creep,  interoperability,  defect  detection, 
backwards  compatibility,  and  the  constant  evolution  of  technol¬ 
ogy.  Despite  these  challenges,  the  body  of  knowledge  for 
managing  these  projects  has  grown  substantially  over  the  last 
few  decades  with  new  innovations  in  our  engineering  models, 
methods,  techniques,  activities,  and  especially  the  automation 
of  tools. 

In  this  issue,  we  have  taken  not  only  a  historical  perspective 
of  past  practices  but  also  highlight  new  innovations  that  con¬ 
tinue  the  advancement  of  project  management  as  a  discipline 
in  its  own  right.  We  begin  this  issue  with  a  collaborate  work 
based  upon  a  study  conducted  at  The  MITRE  Corporation. 

This  article  highlights  the  benefits  of  including  explicit  software 
quality  requirements  at  the  proposal  stage  of  government 
contract  bids,  which  in  turn  would  allow  for  contractor  selection 
to  be  influenced  by  the  use  of  best  practices  in  software  de¬ 
velopment.  We  continue  our  focus  on  quality  with  a  fascinating 
analysis  by  Paul  Croll  in  Quality  Attributes:  Architecting  Sys¬ 
tems  to  Meet  Customer  Expectations.  This  article  emphasizes 
the  importance  of  defining  and  using  a  set  of  quantifiable  qual¬ 
ity  attributes  tied  to  customer  expectations  when  evaluating 
candidate  system  architectures.  With  a  greater  understanding 
of  the  relationship  between  quality  attributes  and  architecture, 
we  can  better  predict  how  candidate  architectures  will  meet 
customer  expectations. 


SMXG  for  sponsoring  this  issue. 

Another  pressing  issue  in  software  project  management  is 
the  increasing  complexity  of  projects  and  the  inherent  difficul¬ 
ties  in  managing  emergent  behavior  in  software  systems.  In 
The  Whole  Is  More  Than  the  Sum  of  Its  Parts:  Understand¬ 
ing  and  Managing  Emergent  Behavior  in  Complex  Systems , 
the  authors  provide  an  overview  of  the  increasing  importance 
of  applying  systems  theory  to  software  as  well  as  explore  some 
speculative  new  methodologies  for  managing  undesirable 
emergent  behavior  in  complex  systems.  To  illustrate  some  of 
the  complexities  we  now  face  in  software  development,  we 
now  turn  to  Developing  a  Model  for  Simplified  Higher  Level 
Sensor  Fusion.  In  this  article,  the  authors  systematically  study 
the  current  difficulties  faced  by  multisensory  data  fusion  pro¬ 
grams  and  ultimately  provide  an  adaptation  of  models  that  can 
be  used  to  provide  an  improved  assessment  while  simplifying 
the  process  needed  to  get  there. 

In  past  issues,  we  have  featured  many  articles  that  provide 
practical  guidance  to  improve  the  quality  of  Earned  Value  Man¬ 
agement  (EVM)  information  and  highlighted  the  value  of  such 
data  in  managing  a  project.  In  Basing  Earned  Value  on  Tech¬ 
nical  Performance ,  Paul  Solomon  readdresses  the  topic  and 
proposes  new  solutions  to  further  enhance  the  value  of  EVM. 
Continuing  down  the  path  of  accurate  and  reliable  information, 
we  need  to  look  no  further  than  William  Roetzheim’s  work  in 
Core  Estimating  Concepts.  This  article  reveals  that  beneath 
the  myriad  of  domain-specific  estimation  tools  available  lies  a 
set  of  core  estimation  concepts  that  can  provide  a  framework 
for  building  new  models  for  your  specific  needs.  We  con¬ 
clude  our  set  of  articles  by  stressing  the  importance  of  peer 
reviews  in  Statistical  Tune-Up  of  the  Peer  Review  Engine 
to  Reduce  Escapes.  In  this  article,  Tom  Lienhard  identifies 
defects  passing  undetected  through  peer  reviews  as  a  major 
source  of  rework  as  a  major  problem  and  proposes  innovative 
improvements  to  the  peer  review  process.  As  always,  be  sure 
not  to  miss  David  Cook’s  humorous,  yet  insightful,  look  back  at 
hard-learned  lessons  to  writing  good  software. 

As  we  begin  the  new  year,  we  are  also  beginning  the  25th 
year  of  crosstalk  publication  as  well.  I  would  like  to  take  a 
moment  to  express  my  sincere  thanks  to  everyone  for  making 
such  an  accomplishment  possible.  To  our  co-sponsors,  we 
thank  you  for  your  generous  support  and  active  involvement 
in  providing  an  information  and  educational  resource  to  the 
software  industry.  To  the  authors,  we  truly  appreciate  all  of 
your  time  and  effort  in  sharing  such  valuable  information  to  the 
software  community.  To  our  readers,  thank  you  for  your  contin¬ 
ued  support  and  hope  that  we  continue  to  exceed  expectations 
by  publishing  the  highest  quality  articles. 

From  all  of  us  at  crosstalk,  we  wish  you  the  best  for  the 
new  year! 

Justin  T.  Hill 

Publisher,  crosstalk 
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Applying  the  Fundamentals  of 
Quality  to  Software  Acquisition 

Steve  Bygren,  The  MITRE  Corporation 
Greg  Carrier,  The  MITRE  Corporation 
Tom  Maher,  The  MITRE  Corporation 
Patrick  Maurer,  The  MITRE  Corporation 
David  Smiley,  The  MITRE  Corporation 
Rick  Spiewak,  The  MITRE  Corporation 
Christine  Sweed,  The  MITRE  Corporation 


How  Do  You  Measure  Software  Quality? 

Software  quality  as  an  outcome  is  best  measured  by  the  num¬ 
ber  of  defects  encountered  after  development  is  complete  as  the 
numerator,  divided  by  the  “size”  of  the  software  as  the  denomina¬ 
tor.  One  could  also  argue  that  if  two  different  products  were  to  be 
compared,  some  sort  of  “difficulty  factor”  could  be  applied,  as  well 
as  references  to  the  software  language  or  development  environ¬ 
ment  employed,  e.g.,  assembly  code  versus  high  order  languages, 
or  object-oriented  versus  functional  languages,  etc. 

Metrics  exist  which  can  be  used  to  estimate  the  potential 
defects  in  code.  These  are  based  on  the  use  of  function  points 
as  the  measure  of  “size.”  Function  points  can  also  be  (loosely) 
correlated  with  the  commonly  used  measurement,  SLOC. 


Abstract.  Historically,  software  developed  under  government  contracts  often 
does  not  stand  up  under  real-world  use,  and  defects  frequently  result  in  cost  and 
schedule  overruns.  While  proposed  development  activities  from  contractors  com¬ 
monly  list  measures  to  improve  quality,  these  descriptions  cannot  be  used  to  select 
a  winning  bidder  if  they  are  not  part  of  the  evaluation  criteria.  By  making  software 
quality  requirements  explicit  at  the  proposal  stage,  contractor  selection  can  be 
influenced  by  criteria  based  on  best  practices  in  software  development. 

If  we  want  to  improve  the  quality  of  our  software,  a  “Qual¬ 
ity  in  Depth”  approach  is  needed-introducing  quality  related 
measures  at  every  stage  of  software  acquisition.  In  a  previ¬ 
ous  article,1  one  of  the  authors  provided  recommendations  for 
improving  software  quality  at  the  construction  phase.  This  article 
discusses  how  to  apply  these  same  principles  to  the  source 
selection  process. 

In  order  to  find  a  way  to  include  software  practices  as  selec¬ 
tion  criteria,  the  authors  set  out  to  identify  and  recommend 
changes  to  Sections  L  and  M  of  a  government  Request  for 
Proposal  (RFP)  or  Instructions  for  Proposal  Preparation  (IFPP) 
and  Evaluation  Criteria  (EC)  in  an  attempt  to  improve  software 
and  system  quality.  These  changes  will  enable  selection  teams 
to  identify  contractors  whose  software  development  processes 
and  compliance  with  software  quality  standards  are  more  likely 
to  produce  the  desired  results. 

1.  Background 

What  Is  Software  Quality? 

Quality  is  often  thought  of  as  an  absence  of  defects.  With 
many  software  products  however,  “defect”  does  not  adequately 
describe  the  range  of  phenomena  that  affect  software  quality  as 
perceived  by  the  customers,  end  users  and  other  stakeholders. 
Using  Crosby’s  philosophy,2  we  define  the  term  “software  quality” 
to  mean  conformance  to  the  requirements  of  the  software  prod¬ 
uct’s  users  and  other  stakeholders.  The  more  closely  a  software 
product  conforms  to  these  requirements,  the  higher  its  quality. 

We  are  particularly  interested  in  software  quality  as  it  affects 
the  acquisition  process  for  defense  related  software.  While 
end  user  requirements  are  of  prime  importance,  poor  software 
development  and  quality  monitoring  practices  in  early-  and 
mid-stage  acquisition  can  result  in  failure  to  provide  the  desired 
results.  These  failures  range  from  unwanted  or  missing  features 
to  cost  and  schedule  overruns  to  critical  flaws  in  system  security 
or  reliability. 


2.  Approach 

This  article  is  the  outcome  of  a  study  the  authors  conducted 
at  MITRE.  Our  approach  was  to  gather  information  from  Subject 
Matter  Experts  (SMEs),  contracting  officers,  and  acquisition 
experts  for  recommendations  for  additions  to  proposal  docu¬ 
ments.  Part  of  this  study  was  conducted  through  interviews  and 
SME  e-mail  group  lists.  Reference  materials  from  the  Air  Force 
and  Navy  were  found  which  provided  recommendations  from 
prior  work  [1,2].  We  then  adapted  the  suggestions  to  Sections 
L  and  M  to  more  thoroughly  describe  software  quality  related 
criteria  for  source  selection.  Some  of  these  criteria  are  aimed 
at  the  technical  evaluation  team,  while  some  can  be  used  by 
cost  evaluators  and  past  performance  evaluators  as  well  as  the 
technical  team. 

3.  Recommendations  for  Section  L 
(Instructions  for  Proposal  Preparation) 

1.  The  offeror’s  proposal  shall  include  a  proposed  Software 
Development  Plan  (SDP)  which  describes  their  approach  to 
software  development,  to  include  the  tools,  techniques  and  stan¬ 
dards  to  be  used  for  development,  unit  testing  and  component 
testing;  integration  tools  and  techniques  (including  configuration 
management)  used  to  ensure  the  integrity  of  system  builds;  the 
number  and  type  of  reviews  that  are  part  of  the  development 
process;  and  the  methods  and  tools  used  to  manage  defect 
reports  and  analysis,  including  root  cause  analysis  as  necessary. 
The  proposed  SDP  will  form  the  basis  for  a  completed  SDP 

to  be  available  after  contract  award  as  a  Contract  Deliverable 
Requirements  List  (CDRL)  item,  subject  to  government  review 
and  approval. 

2.  The  offeror  shall  describe  their  plan  for  effective  code 
reuse  in  order  to  minimize  the  amount  of  new  code  to  be  devel¬ 
oped.  Reused  code  can  come  from  any  origin,  including  previous 
efforts  by  the  offeror  or  as  provided  by  the  Government  in  the 
bidders’  library. 

3.  The  offeror  shall  provide  a  Basis  of  Estimate  (BOE)  de¬ 
scribing  the  rationale  for  the  proposed  staffing.  The  detail  of  the 
BOE  shall  include  labor  hours  for  each  labor  category  (e.g.,  sys¬ 
tem  engineering  staff  versus  software  engineering  staff)  for  the 
identified  tasks  in  the  Work  Breakdown  Structure  as  it  relates  to 
the  Statement  of  Work  (SOW). 

4.  The  offeror  shall  describe  the  process  for  orientation  and 
training  for  all  project  employees  (e.g.  certification  and  training 
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in  software  best  practices  including  information  assurance  and 
risk  management). 

5.  The  offeror  shall  describe  related  systems  experience, 
including  a  description  of  previous  experience  developing 
software  of  the  same  nature,  and  a  description  of  the  extent  to 
which  personnel  who  contributed  to  these  previous  efforts  will 
be  supporting  this  effort. 

6.  The  offeror  shall  describe  proposed  development  prac¬ 
tices.  For  example,  if  spiral/incremental  development,  they  shall 
describe  the  number,  duration,  and  scope  of  spirals,  as  well  as 
how  the  use  of  your  approach  would  result  in  improved  product 
quality  and  user  satisfaction  over  time.3 

7.  The  offeror  shall  provide  an  Integrated  Master  Schedule 
(IMS)  and  accompanying  narrative  that  describes  all  significant 
program  activities  that  are  aligned  with  the  proposed  program 
staffing  profile.  Include  a  timeline  for  completion  of  each  activity 
identified  in  the  proposed  program.  Provide  details  that  clearly 
describe  the  purpose  for  and  importance  of  key  activities.  Iden¬ 
tify  all  critical  path  elements  and  key  dependencies. 

4.  Recommendations  for  Section  M 
(Evaluation  Criteria) 

The  proposed  SDP  shall  show  a  complete  and  comprehen¬ 
sive  software  development  process,  which  incorporates  best 
practices  as  well  as  standards  such  as  IEEE  1 2207-2008. 

The  contractor  will  be  evaluated  based  on  how  their  processes, 
as  described  in  the  SDP,  incorporate  the  use  of  software  best 
practices. 

Evaluation  criteria  related  to  the  SDP  include  the  following: 

•  The  number  and  type  of  peer  reviews. 

•  The  use  of  automated  unit  testing  including  test 
coverage  requirements. 

•  The  use  of  automated  syntax  analysis  tools  and  adherence 
to  the  rules  incorporated  by  them.4 

•  The  comprehensiveness  of  integration  and  test  methods, 
including  continuous  integration  tools  if  used. 

•  The  use  of  readiness  requirements  such  as  unit  test  and 
syntax  analysis  for  code  check-in. 

•  Configuration  management  and  source  code  control  tools 
and  techniques. 

•  The  extent  to  which  root  cause  analysis  of  defects  is  part  of 
the  development  process. 

•  The  selection  of  software  source  code  to  be  reused, 
replaced  or  rewritten  from  previous  implementations  or  other 
origins,  including  a  description  of  how  it  will  be  ensured  that 
reused  code  meets  or  is  brought  up  to  the  same  standards 

as  newly  developed  code.  Risks  associated  with  reused 
software  shall  also  be  discussed.  Such  software  shall  include 
government  rights  to  the  source  code. 

The  IMS  and  accompanying  narrative  will  be  evaluated  for  level 
of  detail  and  relevance  of  significant  program  activities,  degree 
of  alignment,  the  proposed  program  staffing  profile,  and  integra¬ 
tion  of  the  proposed  SDP  into  the  IMS.  Additionally,  critical  path 
elements  and  key  dependencies  will  be  assessed  for  relevance, 
completeness  and  the  manner  and  level  of  risk  containment. 


Parameter/rating 

Unacceptable 

Marginal 

Acceptable 

Superior 

The  number  and  type  of 
peer  reviews 

none 

1  (any) 

2  (design,  code) 

3  or  more 
(requirements, 
design,  code,  test) 

The  use  of  automated 
unit  testing  including  test 
coverage  requirements 

none 

unit  tests 
written  after 
manual 
testing  or 
only  on 
selected 
code 

automated  tests  75% 
code  coverage  on 
new  or  modified  code 

automated  tests  85% 
or  more  code 
coverage  on  all 
delivered  code.  The 
use  of  Test  Driven 
Development. 

The  use  of  automated 
syntax  analysis  tools  and 
adherence  to  the  rules 
incorporated  by  them 

none 

used 

selectively  or 
with  heavily 
modified 
rules 

used  consistently  with 
standard  rules 

additional  rules  or 
tools  specific  to 
security  analysis 

The  comprehensiveness 
of  integration  and  test 
methods  including 
continuous  integration 
tools  if  used 

ad-hoc 

formal 
integration 
and  test 

automated  processes 
applied  periodically 

continuous 
integration  including 
syntax  analysis  and 
unit  tests 

The  use  of  readiness 
requirements  such  as 
unit  test  and  syntax 
analysis  for  code  check¬ 
in 

none 

individual 

manual 

testing 

integrated  testing  by 
developer 

automated  part  of 
check-in  and 
continuous 
integration  process 

Configuration 
management  and  source 
code  control  tools  and 
techniques 

manual/paper  trail 

by  individual 
developer 

system-wide 

repository 

managed  tool  with 

pre-check-in 

requirements 

The  extent  to  which  root 
cause  analysis  of 
defects  is  part  of  the 
development  process 

none 

“red-team” 

only 

serious  defects 

routine  periodic 
analysis  of  defect 
pool 

The  selection  of  software 
source  code  to  be 
reused,  replaced,  or  re¬ 
written  from  previous 
implementations 

none  or  no 

response 

replacement 

with 

contractor’s 

previous 

work 

rework  of  selected 
items  showing  good 
knowledge  of  base 
software 

innovative  approach 
to  maximum  reuse 
and  modernization 

Table  1.  Sample  Rating  Scale  for  SDP  Evaluation  Criteria5 


5.  Incorporating  Software  Quality  Measures 
in  Contracts 

The  contract  development  process  includes  several  steps 
at  which  information  can  be  gathered  and  requirements  set  to 
include  software  quality  as  a  measure  of  vendor  performance. 

Sections  L  &  M  or  equivalent  from  the  RFP 

»  Add  software  quality  measures  as  a  discriminating  factor 
in  selecting  the  contractor 
»  Enumerate  expectations  in  this  area: 

•  Types  of  methods  used 

•  Evidence  to  be  provided 

Technical  Requirements  Document,  Statement  of  Objectives, 
and  SOW 

Add  requirements  in  the  form  of  deliverable  items— as  CDRLs 
or  Data  Accession  List  items  as  appropriate.  Examples  include 
the  following: 

»  Output  of  automated  unit  tests  showing  code  coverage 
at  or  above  required  minimum. 

»  Output  of  automated  syntax  analysis  showing 
conformance  to  pre-determined  rules. 

»  Evidence  of  accomplishing  required  peer  reviews. 

»  Itemized  list  of  tools  with  version  numbers  used  to 
produce  output  from  each  source  module. 

»  Programmer’s  reference  manual  with  examples. 
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»  Interface  definitions. 

»  List  of  all  software  components  with  the  following 
information: 

•  Purpose  and  function. 

•  Interfaces  provided. 

•  Language/version  for  each  module. 

•  Complete  source  code. 

»  Source  from  architectural  design  tool  where  available. 

»  Use  cases  (text  and  diagrams). 

»  Class  diagrams  where  applicable. 

»  Complete  list  of  any  third-party  components  with  version 
numbers. 

»  Contact  information  for  any  outside  dependencies. 

»  Build  procedures,  including  documentation  for  building  all 
software  components  from  source  code. 

»  Test  procedures-including  any  automated  unit  tests  with 
source  code,  test  scripts. 

6.  Rationale  for  Incorporating  Recommended 
RFP  Language 

The  recommended  RFP  language  was  derived  by  the  authors 
from  a  variety  of  sources  including  MITRE  acquisition  subject 
matter  experts,  existing  guidance  documents  from  the  Navy  and 
Air  Force,  and  also  from  the  authors’  experience.  We  have  tried 
to  provide  a  succinct  rationale  as  to  why  the  language  asks  for 
specific  information  from  the  contractor  in  the  RFP: 

The  SDP  is  a  maturity  indicator  of  the  bidder’s  development 
process.  By  evaluating  this,  and  then  putting  its  provisions  under 
contract,  it  becomes  possible  to  select  a  contractor  on  the  basis 
of  development  methodology  and  then  obligate  them  to  perform 
as  proposed. 

Automated  unit  tests  and  comprehensive  peer  reviews  are 
widely  used  best  practices.  Capers  Jones6  has  noted  that  these 
are  among  the  required  steps  to  achieve  effective  defect  removal. 

Continuous  Integration  (Cl)  often  includes  the  automated 
invocation  of  tests  and  code  analysis  during  the  build  process. 

Cl  and  static  analysis  expose  problems  earlier  in  the  develop¬ 
ment  process.  The  earlier  problems  are  discovered,  the  lower 
the  cost  to  resolve. 

Root  cause  analysis  prevents  the  introduction  of  defects 
and  is  a  recognized  best  practice  in  all  approaches  to  process 
improvement.  It  is  a  CM  Ml®  Level  5  practice  area.  Prevention  is 
more  cost  effective  than  detecting  and  fixing  defects  after  they 
are  introduced. 

The  BOE  helps  the  evaluator  understand  the  bidder’s  cost 
to  compare  against  industry  averages  and  government  cost 
models.  By  examining  proposed  labor  categories,  this  can  be 
checked  against  predicted  labor  distributions  from  government 
cost  models  as  well. 

The  IMS  can  be  checked  for  alignment  with  required  mile¬ 
stone  dates,  and  it  supports  an  independent  estimate. 

7.  Guidance  for  Evaluation  Team  Experience 

The  government’s  evaluation  team  must  have  relevant  software 
engineering  experience.  The  experience  should  cover  the  full  life 
cycle  of  software  development  from  design  to  development,  inte¬ 
gration,  testing,  and  delivery.  If  the  proposal  is  seeking  a  particular 


style  of  development  methodology  (e.g.,  waterfall,  spiral/incre¬ 
mental,  agile),  then  the  evaluation  team  should  have  experience  in 
that  methodology  in  order  to  evaluate  the  RFP  response. 

Since  a  significant  portion  of  the  suggested  contract  language 
relates  to  software  quality  monitoring,  the  evaluators  should  be 
familiar  with  unit  testing,  peer  reviews,  Cl,  static  code  analysis, 
and  metrics.  Finally,  evaluators  should  have  some  knowledge  of 
various  practices  and  approaches  of  applying  these  techniques, 
for  example,  when  it  comes  to  test-driven  development. 

The  field  of  software  engineering  is  diverse.  It  is  insufficient 
to  simply  have  general  software  engineering  experience  on  the 
evaluation  team  without  further  having  experience  in  the  appli¬ 
cable  domain(s).  Examples  of  these  domains  include  real-time/ 
embedded,  kernel/operating  systems,  numerical/digital  signal 
processing,  web  applications,  SOA,  information  retrieval/search, 
security,  and  human-computer  interface. 

Finally,  the  evaluation  team  should  have  an  understanding  of 
the  CM  Ml  process  and  rating  criteria. 

8.  Guidance  for  Evaluating  Technical  Responses 

The  recommended  contract  language  in  this  article  includes 
Section  M  of  the  RFR  also  appearing  as  Evaluation  Criteria.  The 
language  is  not  very  specific  so  as  to  elicit  responses  that  are  more 
original  than  simply  claiming  to  do  a  long  list  of  things  that  the  gov¬ 
ernment  is  checking  for.  In  this  section,  we  discuss  more  specific 
guidance  for  the  evaluation  team  in  evaluating  the  responses. 

In  advance,  the  team  should  define  objectives  that  are  sought 
after  and  then  define  measurable  criteria.  The  more  objective  the 
criteria,  the  better,  though  it  is  recognized  that  coming  up  with  this 
criteria  can  be  a  challenge.  After  defining  criteria,  they  are  priori¬ 
tized  and  then  weighted  in  a  scheme  the  team  deems  appropriate. 

Some  general  evaluation  tips  are  as  follows: 

•  If  key  staff  are  identified  in  the  proposal,  how  likely  are  they 
to  be  available  during  contract  execution? 

•  In  reference  to  quality  assurance  processes,  does  the  pro¬ 
posal  language  favor  or  at  least  mention  “empowerment”  of  the 
quality  assurance  team  over  engineering  processes? 

•  Regarding  the  contractor’s  approach  to  automated  unit 
testing:  Does  the  contractor  require  that  unit  tests  be  passed 
and  cover  a  reasonable  percentage  of  code  before  code  can  be 
checked  in?  Does  the  contractor  use  test-driven  development? 

•  Regarding  the  contractor’s  approach  to  automated  syntax 
analysis:  Does  the  contractor  require  that  syntax  analysis  be 
performed  and  that  all  required  rules  are  followed  before  code 
can  be  checked  in? 

•  Regarding  development  build  and  integration:  Does  the  con¬ 
tractor  use  an  automated  build  process  that  incorporates  syntax 
analysis  and  automated  unit  testing? 

You  can  expect  that  the  response  is  going  to  claim  appraisal 
at  a  specific  CM  Ml  maturity  level  (commonly  at  least  level  3). 
This  can  be  verified  with  the  Appraisal  Disclosure  Statement 
(ADS)  document.  Another  source  is  the  Standard  CM  Ml  Ap¬ 
praisal  Method  for  Process  Improvement  (SCAMPI).  For  the 
larger  contractors,  particularly  when  work  is  further  sub-con¬ 
tracted  out,  look  for  further  CM  Ml  level  compliance  information 
on  the  specific  division/unit  and  sub-contractor(s)  as  applicable. 
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9.  Development  Process 

If  the  proposal  declares  that  a  development  process  will  be 
used  that  will  involve  multiple  iterations/spirals/increments 
(which  is  standard  practice),  then  the  evaluation  team  should 
look  for  further  details  on  the  process  to  include  the  following: 

•  What  is  the  duration  and  scope  of  each  increment? 

•  Are  lessons  and  obstacles  from  one  increment  reviewed  for 
improvement  to  a  subsequent  increment? 

•  Is  user  (customer)  feedback  interaction  only  up  front  or  do 
most  increments  incorporate  this?  And  how  is  that  feedback 
prioritized? 

•  Are  multiple  increments  planned  in  sufficient  detail,  or  are 
only  the  present  and  possibly  next  increment  planned? 

10.  Software  Engineering 

One  key  thing  to  look  for  in  a  proposal  is  to  what  degree  the 
contractor  has  experience  in  the  technology  the  RFP  calls  for 
them  to  deliver.  The  more  complex  the  system,  the  more  impor¬ 
tant  applicable  contractor  experience  is. 

Many  DoD  systems  have  a  degree  of  interoperability  and 
integration  required  of  them.  For  integration  with  particular 
systems,  verify  if  the  contractor  has  experience  with  that  system 
or  has  relationships  with  third  parties  with  integration  capabili¬ 
ties  that  will  be  used.  The  contractor  should  also  participate  in 
applicable  Communities  of  Interest. 

Testing  processes  and  technologies  that  support  them  are 
important.  Look  for  information  on  a  test  plan  or  strategy.  If  the 
proposal  is  serious  about  continuous  integration  and  use  of 
supporting  tools,  then  listing  the  software  to  be  used  for  this  is  a 
promising  sign.  Information  on  how  the  tools  are  used  (e.g.,  by  ex¬ 
ception  and/or  monitored  on  a  periodic  basis-and  what  period)  is 
also  telling.  If  the  proposal  includes  information  on  the  proposed 
system  design,  then  the  evaluators  could  look  to  see  how  “test¬ 
able”  the  design  is,  particularly  as  it  is  incrementally  built. 

11.  Conclusions 

While  it  is  important  to  implement  quality  measures  in 
software  construction,  this  is  undertaken  after  a  contractor  has 
been  selected.  The  authors  recommend  an  in-depth  approach, 
beginning  with  the  process  of  selecting  the  contractor.  It  can  be 
easy  to  overlook  the  importance  of  including  specific  language 
in  the  proposal  documents  in  order  to  be  able  to  select  the 
right  contractor  from  those  responding  to  an  RFP.  In  order  to 
accomplish  this  goal,  it  is  critical  to  specify  the  instructions  in 
Section  L  (or  the  IFPP)  and  the  evaluation  criteria  in  Section 
M  (or  the  EC)  so  that  these  can  be  used  to  assign  strengths  or 
weaknesses  appropriately.  This  is  an  early,  but  often  neglected, 
piece  of  the  puzzle  involved  in  building  quality  software  products 
for  defense  applications. 
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Abstract.  This  paper  addresses  the  use  of  quality  attributes  as  a  mechanism 
for  making  objective  decisions  about  architectural  tradeoffs  and  for  providing 
reasonably  accurate  predictions  about  how  well  candidate  architectures  will  meet 
customer  expectations.  Typical  quality  attributes  important  to  many  current  systems 
of  interest  include  performance,  dependability,  security,  and  safety.  This  paper 
begins  with  an  examination  of  how  quality  attributes  and  architectures  are  related, 
including  some  the  seminal  work  in  the  area,  and  a  survey  of  the  current  standards 
addressing  product  quality  and  evaluation.  The  implications  for  both  the  customer 
and  the  system  developer  of  employing  a  quality-attribute-based  approach  to  ar¬ 
chitecture  definition  and  tradeoff  are  then  briefly  explored.  The  paper  also  touches 
on  the  relationship  of  an  architectural  quality-attribute-based  approach  to  engineer¬ 
ing  process  and  process  maturity.  Lastly  the  special  concerns  of  architecting  for 
system  assurance  are  addressed. 

It  Is  About  the  Architecture 

A  recent  presentation  on  systemic  root  cause  analysis  of 
failures  in  DoD  programs  [1]  pointed  out  that: 

“DoD  operational  test  and  evaluation  results  from  October 
2001  through  September  2006  indicated  that  of  29  systems 
evaluated,  approximately.  50%  were  deemed  ‘Not  Suitable’,  or 
‘partially  Not  Suitable’  and  approximately  33%  were  deemed 
‘Not  Effective’,  or  ‘partially  Not  Effective’.” 

The  presentation  went  on  to  say  that  one  of  the  top  1 0 
emerging  systemic  issues,  from  52  in-depth  program  reviews 
since  March  2004  was  inadequate  software  architectures. 

If  we  are  to  be  successful  in  delivering  systems  that  meet 
customer  expectations,  we  must  start  as  early  as  possible  in 
the  design  process  to  understand  the  extent  to  which  those 
expectations  might  be  achieved.  As  we  develop  candidate 
system  architectures  and  perform  our  architecture  tradeoffs,  it  is 
imperative  that  we  define  and  use  a  set  of  quantifiable  system 
attributes  tied  to  customer  expectations,  against  which  we  can 
measure  success. 

In  2006,  the  National  Defense  Industrial  Association  (NDIA)  con¬ 
vened  a  Top  Software  Issues  Workshop  [2]  to  examine  the  current 
most  critical  issues  in  software  engineering  that  impact  the  acquisi¬ 
tion  and  successful  deployment  of  software-intensive  systems. 

The  workshop  identified  85  issues  for  further  discussion, 
which  were  consolidated  into  a  list  of  the  top  seven.  Of  those 
issues  impacting  software-intensive  systems  throughout  the 
lifecycle,  two  emerged  that  were  focused  specifically  on  the 
relationship  between  software  quality  and  architecture: 

•  Ensure  defined  quality  attributes  are  addressed  in 
requirements,  architecture,  and  design. 

•  Define  software  assurance  quality  attributes  that  can 
be  addressed  during  architectural  tradeoffs. 


As  is  true  in  the  defense  systems  case  above,  most  sys¬ 
tems  we  encounter  today  contain  software  elements  and  most 
depend  upon  those  software  elements  for  a  good  portion  of 
their  functionality.  Modern  systems  architecture  issues  cannot 
be  adequately  addressed  without  considering  the  implications  of 
software  architecture. 

Architecture  and  Quality 

What  is  an  architecture?  IEEE  Std  1471-2000  [3,  34]  defines 
an  architecture  for  software  intensive  systems  as: 

“The  fundamental  organization  of  a  system  embodied  in  its 
components,  their  relationships  to  each  other,  and  to  the  envi¬ 
ronment,  and  the  principles  guiding  its  design  and  evolution.” 

More  recently,  Firesmith  et  al  [4],  their  Method  Framework 
for  Engineering  System  Architectures  (MFESA),  have  defined 
system  architecture  as: 

“The  set  of  all  of  the  most  important,  pervasive,  higher-level, 
strategic  decisions,  inventions,  engineering  tradeoffs,  assumptions, 
and  their  associated  rationales  concerning  how  the  system  meets 
its  allocated  and  derived  product  and  process  requirements.” 

The  authors  believe  that  system  architecture  is  a  major  deter¬ 
minant  of  resulting  system  quality. 

MFESA  instructs  that  architectures  can  be  represented  by 
models,  views,  and  focus  areas.  Models  describe  system  struc¬ 
tures  in  terms  of  their  architectural  elements  and  the  relation¬ 
ships  between  them.  These  descriptions  can  be  graphical  or 
textual  and  include  the  familiar  data  and  control  flow  diagrams, 
entity-relationship  diagrams,  and  UML  diagrams  and  associ¬ 
ated  use  cases.  Views  are  composed  of  one  or  more  related 
architectural  models.  They  use  the  example  of  a  class  view  that 
describes  all  architectural  classes  and  their  relationships.  Focus 
areas  combine  multiple  views  and  models  to  determine  how  the 
architecture  achieves  specific  quality  characteristics. 

What  is  quality  and  what  are  quality  characteristics?  IEEE 
Standard  1 061-1998  [5],  defines  software  quality  as  the  degree 
to  which  software  possesses  a  desired  combination  of  attributes. 

Similarly,  ISO/IEC  9126-1 :2001  [6],  one  of  a  four-part  set  of 
standards  on  software  product  quality,  defines  quality  as: 

“The  totality  of  characteristics  of  an  entity  that  bear  on  its 
ability  to  satisfy  stated  and  implied  needs.” 

The  standard  identifies  a  quality  model  with  six  quality  charac¬ 
teristics:  functionality,  reliability,  usability,  efficiency,  maintainabil¬ 
ity  and  portability.  The  other  three  standards  in  the  91 26-series 
[7,  8,  9]  address  metrics  for  measuring  attributes  of  the  quality 
characteristics  defined  in  ISO/IEC  9126-1. 

It  should  be  noted  that  the  91 26-series  is  being  revised  as 
part  of  the  Software  Product  Quality  Requirements  and  Evalu¬ 
ation  (SQuaRE)  series  of  standards.  ISO/IEC  25010,  Software 
engineering-SQuaRE-quality  model  [10]  is  the  revision  of 
ISO/IEC  9126-1:2001.  ISO/IEC  25010  adds  security  and 
interoperability  to  the  list  of  six  quality  characteristics  defined  in 
ISO/IEC  9126-1 :2001 .  Additionally  ISO/IEC  25030,  Software 
engineering-SQuaRE)-quality  requirements  [1  1]  defines  the 
concept  of  internal  software  quality  as  the  “capability  of  a  set 
of  static  attributes  (including  those  related  to  software  archi¬ 
tecture)  to  satisfy  stated  and  implied  needs  when  the  software 
product  is  used  under  specified  conditions.”  and  the  concept  of 
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software  quality  in  use,  which  is  “the  capability  of  the  software 
product  to  enable  specific  users  to  achieve  specific  goals  with 
effectiveness,  productivity,  safety  and  satisfaction  in  specific 
contexts  of  use.” 

Functional  properties  determine  what  the  software  is  able  to 
do.  Quality  properties  determine  how  well  the  software  performs. 
In  other  words,  the  quality  properties  show  the  degree  to  which 
the  software  is  able  to  provide  and  maintain  its  specified  services. 

ISO/IEC/IEEE  1 5288  [12]  addresses  the  confluence  of  ar¬ 
chitecture  and  quality  in  the  context  of  the  system  lifecycle.  The 
Architectural  Design  Process  (6.4.3)  provides  for  the  creation  of 
design  criteria  for  quality  characteristics  and  the  evaluation  of 
alternative  designs  with  respect  to  those  criteria.  There  is  also 
a  Specialty  Engineering  view  of  the  lifecycle  processes  in  that 
focuses  on  the  achievement  of  product  characteristics  that  have 
been  selected  as  being  of  special  interest. 

Quality  Attribute-based  Approaches 
to  Architecting  Systems 

In  the  seminal  report  on  Quality  Attributes  by  Barbacci  et  al 
[13]  the  authors  indicate  that: 

“Developing  systematic  ways  to  relate  the  software  qual¬ 
ity  attributes  of  a  system  to  the  system’s  architecture  provides 
a  sound  basis  for  making  objective  decisions  about  design 
tradeoffs  and  enables  engineers  to  make  reasonably  accurate 
predictions  about  a  system’s  attributes  that  are  free  from  bias 
and  hidden  assumptions.  The  ultimate  goal  is  the  ability  to 
quantitatively  evaluate  and  trade  off  multiple  software  quality 
attributes  to  arrive  at  a  better  overall  system.” 

Franch  and  Carvallo  [14]  suggest  that  for  an  effective  quality 
model,  the  relationships  between  quality  attributes  must  be 
explicitly  stated  to  understand  potential  attribute  clash  when  de¬ 
fining  software  architectures.  They  posit  three  types  of  relation¬ 
ships  between  attributes: 

•  Collaboration,  in  which  increasing  the  degree  to  which  one 
attribute  is  realized  increases  the  realization  of  another. 

•  Damage,  in  which  increasing  the  degree  to  which  one 
attribute  is  realized  decreases  the  realization  of  another. 

•  Dependency,  in  which  the  degree  to  which  one  attribute  is 
realized,  is  dependent  upon  the  realization  of  at  least  some 
sub-characteristics  of  another. 

For  example,  as  Haggander  et  al  [15]  point  out  using  the 
example  of  a  large  telecommunication  application,  system  archi¬ 
tects  must  balance  multiple  quality  attributes,  such  as  maintain¬ 
ability,  performance  and  availability.  Focusing  solely  on  the  attri¬ 
bute  of  maintainability  often  results  in  poor  system  performance 
and  conversely  focusing  on  performance  and  availability  alone 
may  result  in  result  in  poor  maintainability.  Explicit  architectural 
decisions  can  facilitate  optimization  among  quality  attributes. 

Architectural  Design  and  Tradeoff 

Bass  and  Kazman  [16]  suggest  five  foundational  structures 
that  together  completely  describe  an  architecture  and  that  can 
serve  as  the  basis  for  understanding  the  relationship  of  archi¬ 
tectural  decisions  to  quality  attributes: 


•  Functional  structure  is  the  decomposition  of  the 
functionality  that  the  system  needs  to  support 

•  Code  structure  is  the  code  abstractions  from  which  the 
system  is  built. 

•  Concurrency  structure  is  the  representation  of  logical  con 
currency  among  the  components  of  the  system. 

•  Physical  structure  is  just  that,  the  structure  of  the  physical 
components  of  the  system. 

•  Developmental  structure  is  the  structure  of  the  files  and  the 
directories  identifying  the  system  configuration  as  the 
system  evolves. 

Bass  and  Kazman  [16]  further  suggest  some  likely  relation¬ 
ships  between  the  architectural  structures  described  above 
and  examination  of  the  impact  of  architectural  decisions  upon 
specific  quality  attributes.  They  suggest  for  example  that: 

•  Concurrency  and  physical  structures  are  useful  in 
understanding  system  Performance. 

•  Concurrency  and  code  structures  are  useful  in 
understanding  system  security. 

•  Functional,  code,  and  developmental  structures  are  useful 
in  understanding  system  maintainability. 

Wojcik  et  al  [1  7]  describe  an  Attribute-driven  Design  (ADD) 
method  in  which  the  approach  to  defining  software  architec¬ 
ture  is  based  on  software  quality  attribute  requirements.  ADD 
produces  an  initial  software  architecture  description  from  a  set 
of  design  decisions  that  show: 

•  Partitioning  of  the  system  into  major  computational  and 
developmental  elements. 

•  What  elements  will  be  part  of  the  different  system 
structures,  their  type,  and  the  properties  and  structural 
relations  they  possess. 

•  What  interactions  will  occur  among  elements,  the 
properties  of  those  interactions,  and  the  mechanisms  by 
which  they  occur. 

In  the  very  first  step  in  ADD,  quality  attributes  are  expressed 
as  the  system’s  desired  measurable  quality  attribute  response  to 
a  specific  stimulus.  Knowing  these  requirements  for  each  quality 
attribute  supports  the  selection  of  design  patterns  and  tactics  to 
achieve  those  requirements. 

Kazman  et  al  [18]  describe  an  Architecture  Tradeoff  Analysis 
Method  (ATAM)  that  can  be  used  when  evaluating  an  architec¬ 
ture,  including  those  produced  by  the  ADD  method  above,  in 
order  to  understand  the  consequences  of  architectural  decisions 
with  respect  to  quality  attributes.  As  the  authors  point  out,  ATAM 
is  dependent  upon  quality  attribute  characterizations,  like  those 
produced  through  ADD,  that  provide  the  following  information 
about  each  attribute: 

•  The  stimuli  to  which  the  architecture  must  respond. 

•  How  the  quality  attribute  will  be  measured  or  observed  to 
determine  how  well  it  has  been  achieved. 

•  The  key  architectural  decisions  that  impact  achieving  the 
attribute  requirement. 
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Electrical  Engineers  and  Computer  Scientists 
Be  on  the  Cutting  Edge  of  Software  Development 


The  Software  Maintenance  Group  at  Hill  Air  Force  Base  is  recruiting  civilians  (U.S.  Citizenship  Required). 

Benefits  include  paid  vacation,  health  care  plans,  matching  retirement  fund,  tuition  assistance,  and 
time  paid  for  fitness  activities.  Become  part  of  the  best  and  brightest! 

Hill  Air  Force  Base  is  located  close  to  the  Wasatch  and  Uinta 


ATAM  takes  proposed  architectural  approaches  and  analyzes 
them  based  upon  quality  attributes,  generally  specified  in  terms 
of  scenarios  addressing  stimuli  and  responses.  ATAM  also  iden¬ 
tifies  sensitivity  points  and  tradeoff  points. 

ATAM  describes  stakeholders’  interaction  with  the  system. 
Stakeholders  bring  different  views  to  the  system  and  may 
include  users,  maintainers,  developers,  and  acquirers.  Scenarios 
specify  the  kinds  of  operations  over  which  performance  needs 
to  be  measured,  or  the  kinds  of  failures  the  system  will  have  to 
withstand.  ATAM  uses  three  types  of  scenarios: 

•  Use  case  scenarios,  describing  typical  uses  of  the  system. 

•  Growth  scenarios,  addressing  planned  changes  to 
the  system. 

•  Exploratory  scenarios,  addressing  any  possible  extreme 
changes  that  would  stress  the  system. 

Making  the  Case  for  Architectural  Quality 

How  do  stakeholders  know  that  the  system  will  exhibit 
expected  quality  characteristics?  Firesmith  et  al  [4]  suggest  that 
one  method  is  the  quality  case,  or  more  specifically  for  evaluat¬ 
ing  architectures,  the  architectural  quality  case.  Quality  cases 
consist  of  the  set  of  claims,  supporting  arguments,  and  support¬ 


ing  evidence  that  provide  confidence  that  the  system  will  in  fact 
demonstrate  its  expected  quality  characteristics.  Common  types 
of  quality  cases  include  safety  cases  [19],  and  security  cases 
[20],  and  the  more  generalized  assurance  cases  [21].  Architec¬ 
tural  quality  cases  describe  the  architectural  claims,  supporting 
arguments,  including  architectural  decisions  and  tradeoffs,  archi¬ 
tectural  representations,  and  demonstrations  that  the  architec¬ 
ture  will  exhibit  its  expected  quality  characteristics. 

The  implications  for  both  the  customer  and  the  system 
developer  of  employing  a  quality-attribute-  based  approach  to 
architecture  definition  and  tradeoff,  documented  in  part  by  a 
quality  case,  are  that: 

•  Customer  quality  requirements  will  have  been  distilled 
into  architectural  drivers  [17]  that  will  have  shaped  the 
system  architecture. 

•  Tradeoffs  will  have  been  made  to  optimize  the  realization 
of  important  quality  characteristics,  in  concert  with 
customer  expectations. 

•  The  level  of  confidence  that  the  resultant  architecture  will 
meet  those  expectations  will  be  known. 

•  Customers  will  be  knowledgeable  of  any  residual  risk  they 
are  accepting  by  accepting  the  delivered  system. 
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There  are  also  architectural  implications  regarding  sustain¬ 
ment  of  a  system  over  its  lifecycle.  Croll  [22]  cites  that  with 
respect  to  sustainment,  paying  insufficient  attention  to  sustain¬ 
ment  issues  early  in  the  lifecycle,  including  licensing,  and  prod¬ 
uct  support  can  lead  to  problems  when  commercial  products 
or  components  inevitably  change  or  when  their  suppliers  either 
discontinue  support  or  go  out  of  business.  In  the  hardware  world 
Diminishing  Manufacturing  Sources  and  Materials  Shortage 
(DMSMS)  analyses  are  generally  done  when  integrating  com¬ 
mercial  components  as  part  of  an  approach  for  managing  the 
risk  of  obsolescence  [23].  DMSMS  analyses  focus  on  supplier 
viability  for  the  product  of  interest,  which  could  be  considered  an 
attribute  of  component  maintainability.  Certainly,  such  analyses, 
where  necessary,  should  be  part  of  the  quality  case.  The  speci¬ 
fication  and  realization  of  architectures  which  are  resilient  with 
respect  to  the  substitution  of  alternate  software  components 
can  further  enhance  system  quality  through  the  lifecycle. 

Process  Maturity  Does  Not  Guarantee  Product  Quality 

We  spend  much  time  these  days  focusing  on  the  maturity 
of  our  engineering  processes  and  heralding  process  maturity 
ratings  such  as  those  associated  with  the  CM  Ml®  [24],  for 
development  and  the  ISO  9000  series  [25][26][27],  for  quality 
management  systems,  as  indicators  of  our  ability  to  deliver 
quality  products  -  products  that  meet  the  customer’s  expec¬ 
tations  and  that  continue  to  do  so  throughout  their  lifecycle. 
What  our  customers  have  found,  however,  is  that  often  process 
maturity  does  not  guarantee  product  quality.  This  is  especially 
true  for  the  highly  software  intensive  systems  we  now  build, 
where  performance,  dependability,  and  failure  modes  are  less 
well  understood. 

For  example,  although  the  CMMI  embodies  the  process 
management  premise  that,  the  quality  of  a  system  or  product  is 
highly  influenced  by  the  quality  of  the  process  used  to  develop 
and  maintain  it  [24],  Hefner  [28]  points  out: 

Several  recent  program  failures  from  organizations  claiming 
high  maturity  levels  have  caused  some  to  doubt  whether  CMMI 
improves  the  chances  of  a  successful  project. 

He  goes  on  to  say,  “an  CMMI  appraisal  indicates  the  organiza¬ 
tion’s  capacity  to  perform  the  next  project,  but  cannot  guarantee 
that  each  new  project  will  perform  in  that  way.” 

Understanding  and  Leveraging  a  Supplier’s  CMMI  Efforts:  A 
Guidebook  for  Acquirers  [29]  further  underscores  the  problem 
and  offers  several  cautions  for  acquirers,  with  respect  to  supplier 
claims  of  process  maturity. 

•  A  CMMI  rating  or  CMMI  level  is  not  a  guarantee  of 
program  success. 

•  Organizations  that  have  attained  CMMI  maturity  level 
ratings  do  not  necessarily  apply  those  appraised 
processes  to  a  new  program  at  program  startup. 

•  Organizations  that  claim  CMMI  ratings  are  not  always 
dedicated  to  [maintaining]  process  improvement  [through 
out  the  development  effort]. 


•  Organizations  may  sample  only  a  few  exemplar  programs 
and  declare  that  all  programs  are  being  executed  at  that 
CMMI  level  rating. 

•  Organizations  that  claim  a  high  maturity  level  rating  (level  4 
and  5)  are  not  necessarily  better  suppliers  than  a  level 

3  supplier.  Maturity  levels  4  and  5,  when  compared  across 
different  suppliers,  are  not  created  equal. 

Although  process  maturity  can  in  many  cases  improve  project 
performance  [30],  special  attention  to  the  engineering  processes 
is  required  to  ensure  that  customer  quality  expectations  are  real¬ 
ized  in  resultant  products. 

A  Current  Concern:  Architecting  for  System  Assurance 

Stakeholder  discussion  over  the  last  several  years  has  dem¬ 
onstrated  a  reasonably  consistent  view  of  the  problem  space. 
System  assurance  can  be  viewed  as  the  level  of  confidence 
that  the  system  functions  as  intended  and  is  free  of  exploitable 
vulnerabilities,  either  intentionally  or  unintentionally  designed 
or  inserted  as  part  of  the  system.  The  President’s  Information 
Technology  Advisory  Committee  report  entitled  Cyber  Security: 

A  Crisis  of  Prioritization  [31]  states,  “...  the  approach  of  patching 
and  retrofitting  networks,  computing  systems,  and  software  to 
‘add’  security  and  reliability  may  be  necessary  in  the  short  run 
but  is  inadequate  for  addressing  the  Nation’s  cyber  security 
needs.”  The  report  further  suggests,  “we  simply  do  not  know 
how  to  model,  design,  and  build  systems  incorporating  integral 
security  attributes.” 

As  Croll  points  out  [22],  the  systems  engineering  challenge, 
with  respect  to  assurance,  is  in  integrating  a  heterogeneous  set 
of  globally  engineered  and  supplied  proprietary,  open-source, 
and  other  software;  hardware;  and  firmware;  as  well  as  legacy 
systems;  to  create  well-engineered  integrated,  interoperable,  and 
extendable  systems  whose  security,  safety,  and  other  risks  are 
acceptable-or  at  least  tolerable. 

Baldwin  [32]  underscores  this  challenge  for  DoD  systems  by 
describing  a  vision  for  assurance  in  which  the  requirements  for 
assurance  are  allocated  among  the  right  systems  and  their  criti¬ 
cal  components,  and  such  systems  are  designed  and  sustained 
at  a  known  level  of  assurance. 

The  National  Defense  Industrial  Association  System  Assur¬ 
ance  Guidebook  [33]  describes  practices  in  architectural  design 
that  can  improve  assurance.  The  Guidebook  suggests  some 
general  architectural  principles  for  assurance: 

•  Isolate  critical  components  from  less-critical  components. 

•  Make  critical  components  easier  to  assure  by  making  them 
smaller  and  less  complex. 

•  Separate  data  and  limit  data  and  control  flows. 

•  Include  defensive  components  whose  job  is  to  protect 
other  components  from  each  other  and/or  the 
surrounding  environment. 

•  Beware  of  maximizing  performance  to  the  detriment 
of  assurance. 


12  CrossTalk-Janucry/February  2013 


SOFTWARE  PROJECT  MANAGEMENT  -  LESSONS  LEARNED 


The  Guidebook  also  suggests  using  system  assurance 
requirements,  design  constraints  and  system  assurance  critical 
scenarios  for  architectural  tradeoff  analysis,  and  documenting 
the  results  in  the  assurance  case. 

Summary 

If  we  are  to  be  successful  in  delivering  systems  that  meet 
customer  expectations,  we  must  start  as  early  as  possible  in 
the  design  process  to  understand  the  extent  to  which  those 
expectations  might  be  achieved.  As  we  develop  candidate 
system  architectures  and  perform  our  architecture  tradeoffs,  it 
is  imperative  that  we  define  and  use  a  set  of  quantifiable  quality 
attributes  tied  to  customer  expectations,  against  which  we  can 
measure  success. 

Standards  like  ISO/I  EC  TR  9126,  Parts  1-4,  ISO/I  EC  25010, 
and  ISO/I  EC  2530  can  help  stakeholders  define  quality  at¬ 
tributes  from  both  an  internal  perspective,  useful  for  addressing 
architectural  design,  and  a  quality  in  use  perspective  addressing 
system  realization. 

Methods  have  been  documented  to  aid  in  understanding 
the  relationship  of  architectural  decisions  to  quality  attributes, 
for  defining  software  architecture  is  based  on  software  quality 
attribute  requirements,  and  for  understanding  the  consequences 
of  architectural  decisions  with  respect  to  quality  attributes. 

Architectural  quality  cases  describe  the  architectural  claims, 
supporting  arguments,  including  architectural  decisions  and 
tradeoffs,  architectural  representations,  and  demonstrations  that 
the  architecture  will  exhibit  its  expected  quality  characteristics. 
They  are  extremely  useful  in  providing  customers  with  an  un¬ 
derstanding  of  any  residual  risk  they  are  accepting  by  accepting 
the  delivered  system. 

Several  recent  program  failures  from  organizations  claiming 
high  maturity  levels  have  caused  some  doubt  about  whether 
process  maturity  improves  the  chances  of  a  delivering  a  suc¬ 
cessful  product.  This  is  especially  true  for  the  highly  software 
intensive  systems  we  now  build,  where  performance,  depend¬ 
ability,  and  failure  modes  are  less  well  understood. 

Of  special  concern  these  days  is  architecting  systems  for  sys¬ 
tem  assurance.  Given  our  track  record  in  architecting  systems 
to  meet  assurance  concerns,  guidance  is  needed  to  support 
assurance-specific  architectural  design  and  tradeoff  analysis,  as 
well  as  appropriate  documentation  of  assurance  claims,  argu¬ 
ments,  and  supporting  evidence,  so  that  customers  understand 
the  degree  to  which  the  architecture  mitigates  assurance  risks. 
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The  Whole  Is 
More  Than  the 
Sum  of  Its  Parts: 

Understanding  and  Managing  Emergent 
Behavior  in  Complex  Systems 

Dean  M.  Morris,  Software  Engineering  Consultant 
Kevin  MacG.  Adams,  Ph.D.,  NCSOSE 

Abstract.  How  does  the  project  or  maintenance  manager  control  the  unknown? 
The  unknown  in  this  case  is  the  negative  or  positive  behaviors  or  properties  that 
emerge  from  a  complex  software  system.  The  application  of  systems  theory  to 
software  is  becoming  increasingly  important  as  systems  become  more  complex. 
Looking  at  a  complex  software  system  through  the  lens  of  systems  science 
can  give  the  manager  the  insight  needed  to  understand  and  control  negative  or 
enhance  positive  emergent  behaviors.  This  article  provides  an  overview  of  some  of 
the  key  terms  and  concepts  of  systems  theory,  complexity,  and  emergence.  Both 
the  positive  and  negative  effects  of  emergent  behavior  on  software  systems  are 
considered.  Additionally,  some  speculative  and  new  methodologies  for  managing 
undesirable  emergent  behavior  are  explored. 

Introduction 

Imagine  a  hypothetical  scenario  where  you  were  contracted 
to  manage  the  development  and  maintenance  of  a  new  multi¬ 
server,  web-based  software  system  for  the  Defense  Finance 
and  Accounting  Service.  The  system  was  tested  vigorously  and 
passed  all  acceptance  tests.  After  several  months  of  opera¬ 
tion,  the  system  users  noticed  increasing  lag  in  the  system 
until  it  finally  locked  up.  Tests  of  the  individual  components  of 
the  system  indicated  there  were  no  problems.  It  only  displayed 
the  lock-up  problem  when  the  entire  system  was  online  and 
operating.  So,  what  happened?  Why  did  the  system  exhibit  this 
negative  behavior  only  after  being  in  operation  for  some  time? 

A  real-world  variation  of  this  scenario  at  another  organization 
was  presented  by  Mogul  [1].  After  much  troubleshooting  by  the 
maintenance  team,  it  was  diagnosed  that  the  database  server 
load  balancer  was  set  incorrectly.  As  data  was  being  received, 
routed,  and  stored  in  the  databases,  the  databases’  response 
time  increased.  The  unexpected  effect  was  the  system  load 
balancer  interpreted  the  increased  database  delays  as  a 
failure.  After  the  timing  expectations  of  the  load  balancer  were 
lowered  (i.e.  the  expected  response  time  from  the  servers  was 
increased),  the  system  functioned  well. 

In  hindsight,  it  was  concluded  that  the  behavior  of  the  load  bal¬ 
ancer  was  totally  unexpected  as  it  only  manifested  itself  when  the 
entire  system  was  operating.  This  type  of  unexpected  or  emer¬ 
gent  behavior  in  a  complex  system  is  one  of  the  key  concepts 


of  systems  theory  [2].  As  software  systems  have  become  much 
more  prevalent  in  government,  commercial,  and  peoples’  daily 
lives,  so  too  has  the  complexity  of  the  systems  that  support  them. 
The  knowledge  and  application  of  systems  theory  and  methodolo¬ 
gies  is  becoming  increasingly  important  for  the  management  and 
maintenance  of  today’s  complex  software  systems. 

A  Brief  Primer  of  Systems  Theory  and  Emergence 

Systems  theory  [3]  provides  the  underlying  theoretical  founda¬ 
tion  for  understanding  systems,  and  as  such,  serves  as  the  foun¬ 
dation  for  the  purposeful  engineering  for  all  complex  systems. 
Knowledge  of  the  systems  theory  axioms  is  essential  for  the 
modern  software  project  or  maintenance  manager.  While  under¬ 
standing  all  the  basic  concepts  of  systems  theory  is  important 
in  software  systems,  the  concept  of  emergence  or  emergent 
behavior  (both  positive  and  negative)  is  paramount.  By  its  nature, 
a  software  system  usually  only  exhibits  emergent  behavior(s) 
after  the  system  has  been  accepted  and  transitioned  to  the 
software  maintenance  phase.  Thus,  the  thrust  of  this  article  is 
toward  managing  the  maintenance  of  complex  software  systems 
with  the  potential  to  exhibit  emergent  behaviors. 

Systems  Theory 

Systems  theory  is  a  unified  system  of  propositions,  linked 
with  the  aim  of  achieving  an  understanding  of  systems,  while 
invoking  improved  explanatory  power  and  predictive  ability.  It  is 
precisely  this  group  of  propositions  that  enables  thinking  and  ac¬ 
tion  with  respect  to  systems  [3].  A  theory  does  not  have  a  single 
proposition  that  defines  it,  but  is  a  population  of  propositions 
(a  model)  that  provides  a  skeletal  structure  for  the  explanation 
of  real-world  phenomena.  The  relationship  between  theory  and 
its  propositions  is  not  a  direct  relationship.  It  is  indirect,  through 
the  intermediary  of  the  axioms,  where  the  links  in  the  theory 
represent  the  correspondence  through  similarity  to  the  empirical, 
real-world  system.  Figure  1  depicts  these  relationships. 


Figure  1 :  Propositions,  Axioms,  Theory  and  the  Real 
World  System 
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(C)  (d) 


Figure  3:  Snapshots  from  an  Ant  Colony  Simulation  [7] 


Systems  theory  provides  explanations  for  real  world  systems. 
The  explanations  increase  our  understanding  and  provide 
improved  levels  of  explanatory  power  and  interpretation  for  the 
real  world  systems  we  encounter.  Our  view  of  systems  theory  is 
a  model  of  linked  axioms  that  are  represented  through  similar¬ 
ity  to  the  real  system  [4].  Figure  2  is  a  model  of  the  axioms  of 
systems  theory.  The  axioms  presented  in  Figure  2  are  called  the 
theorems  of  the  system  or  theory  [5]  and  are  the  select  set  of 
propositions,  presumed  true  by  systems  theory,  from  which  all 
other  propositions  in  systems  theory  are  deducible. 


The  axioms  of  systems  theory  [6]  are  as  follows: 

•  The  Centrality  Axiom  states  that  central  to  all  systems  are 
two  pairs  of  propositions;  emergence  and  hierarchy,  and  com¬ 
munication  and  control. 

•  The  Contextual  Axiom  states  that  system  meaning  is 
informed  by  the  circumstances  and  factors  that  surround  the 
system.  The  contextual  axiom’s  propositions  are  those  which 
give  meaning  to  the  system  by  providing  guidance  that  enable 
an  investigator  to  understand  the  set  of  external  circumstances 
or  factors  that  enable  or  constrain  a  particular  system. 

•  The  Goal  Axiom  states  that  systems  achieve  specific 
goals  through  purposeful  behavior  using  pathways  and 
means.  The  goal  axiom’s  propositions  address  the  pathways 
and  means  for  implementing  systems  that  are  capable  of 
achieving  a  specific  purpose. 

•  The  Operational  Axiom  states  that  systems  must  be  ad¬ 
dressed  in  situ,  where  the  system  is  exhibiting  purposeful 
behavior.  The  operational  axiom’s  propositions  provide  guid¬ 
ance  to  those  that  must  address  the  system  in  situ,  where  the 
system  is  functioning  to  produce  behavior  and  performance. 

•  The  Viability  Axiom  states  that  key  parameters  in  a  system 
must  be  controlled  to  ensure  continued  existence.  The  viability 
axiom  addresses  how  to  design  a  system  so  that  changes  in 
the  operational  environment  may  be  detected  and  affected  to 
ensure  continued  existence. 

•  The  Design  Axiom  states  that  system  design  is  a  pur¬ 
poseful  imbalance  of  resources  and  relationships.  Resourc¬ 
es  and  relationships  are  never  in  balance  because  there  are 
never  sufficient  resources  to  satisfy  all  of  the  relationships 
in  a  systems  design.  The  design  axiom  provides  guidance 
on  how  a  system  is  planned,  instantiated,  and  evolved  in  a 
purposive  manner. 

•  The  Information  Axiom  states  that  systems  create,  possess, 
transfer,  and  modify  information.  The  information  axiom  provides 
understanding  of  how  information  affects  systems. 

Emergence 

Central  to  the  discussion  of  system  theory  is  the  centrality 
axiom  and  the  principles  of  emergence  and  hierarchy.  Hier¬ 
archy  and  emergence  contribute  to  complexity  because  new 
and  interesting  properties  that  cannot  be  found  in  the  parts 
emerge  and  add  a  whole  new  dimension  to  understanding  [2]. 
The  father  of  modern  systems  theory,  Ludwig  von  Bertalanffy 
explains  that  the  meaning  of  the  somewhat  mystical  expres¬ 
sion,  “The  whole  is  more  than  the  sum  of  its  parts,”  is  simply 
that  constitutive  characteristics  are  not  explainable  from  the 
characteristics  of  isolated  parts.  The  characteristics  of  the 
complex,  therefore,  compared  to  those  of  the  elements,  appear 
as  new  or  emergent  [7]. 

However,  Odell  [8]  noted  that  complex  systems  do  not  have 
to  be  complicated  to  display  emergent  behavior.  In  fact,  the 
agents  (i.e.,  the  elements)  of  the  system  can  all  be  homog¬ 
enous,  follow  a  simple  set  of  rules,  and  still  exhibit  emergent 
properties.  To  illustrate  the  point,  Odell  [8]  described  an  ant 
colony  computer  simulation  where  each  ant  agent  behaved  by 
the  following  rules: 
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1 .  Wander  randomly. 

2.  If  food  is  found,  take  a  piece  back  to  the  colony  and  leave 
a  trail  of  pheromones  that  evaporate  over  time;  then  go 
back  to  rule  1 . 

3.  If  a  pheromone  trail  is  found,  follow  it  to  the  food  and  then 
go  to  rule  2. 

Figure  3  shows  the  simulation  display  at  four  stages.  The 
anthill  is  represented  by  the  purple  circle  in  the  center  and  the 
three  blue  dots  are  the  food  piles.  The  ants  are  the  small  red 
dots  and  their  pheromone  trails  are  the  white  and  green  areas. 

Notice  that  in  Figure  3.a,  the  ants  have  individually  begun 
moving  away  from  the  colony  in  a  random  way.  Later  in  the 
sequence,  in  Figure  3.b,  some  ants  have  located  the  food  on  the 
right  and  have  started  marking  their  path  back  to  the  anthill  with 
pheromones  as  they  carry  bits  of  food.  By  the  time  of  Figure  3.d, 
the  entire  right  food  pile  has  been  moved  to  the  colony  and  they 
are  rapidly  consuming  the  other  two  piles  of  food. 

Relating  back  to  systems  theory,  while  the  ants  individually 
behave  by  a  simple  rule  set,  collectively  as  a  colony  system,  they 
act  in  a  complex  way.  They  communicate  with  each  other  via  the 
pheromone  trails,  while  the  pheromones  also  serve  to  exert  over¬ 
all  system  control.  That  is,  when  an  ant  encounters  a  pheromone 
trail,  it  is  obligated  to  follow  it  to  the  food,  get  some  food,  and 
return  to  the  anthill  marking  the  path  with  more  pheromones.  The 
resulting  emergent  property  for  the  ant  colony  system  is  a  full 
food  storage  area.  This  emergent  property  cannot  be  discerned 
by  observing  the  behavior  (via  the  simple  rule  set)  of  individual 
ants.  Only  once  the  ants  begin  interacting  in  an  environment 
where  there  is  food  and  a  storage  area  (i.e.  the  colony)  does  the 
emergent  property  of  food  storage  become  evident. 

Effects  of  Complexity  and  Emergence 
on  Software  Systems 

This  section  moves  systems  theory  into  the  realm  of  software 
systems.  Before  getting  into  some  positive  and  negative  impli¬ 
cations  of  complexity  and  emergence  within  software  systems, 
a  more  abstract  view  of  the  software  and  its  stakeholders  will 
be  discussed  in  a  systems  theory  context. 

Evolution  of  Software 

Rajlich  and  Bennett  [9]  developed  a  versioned,  staged  model 
for  software  maintenance  because  they  believed  that  the  more 
traditional  models  of  maintenance  did  not  accurately  capture  the 
evolutionary  nature  of  the  software  lifecycle.  Rajlich  and  Bennett’s 
[9]  maintenance  model  consists  of  the  following  five  stages: 

1.  Initial  development:  Not  maintenance  yet. 

2.  Evolution:  Significant  changes  may  be  made  to  a  given 
version  to  meet  changing  user  needs.  The  experience  of 
the  development  team  and  an  adaptable  architecture  are 
being  leveraged  to  accommodate  the  major  changes. 

This  stage  is  iterative  for  the  given  version. 

3.  Servicing:  As  team  experience  and  knowledge  for  the 
version  is  lost  and  the  code  starts  to  decay;  only  minor 
updates  are  made  to  the  system.  This  stage  is 

also  iterative. 


4.  Phaseout:  No  more  updates  are  performed  as  the  system 
continues  to  operate. 

5.  Closedown:  The  system  is  retired  from  service. 

In  this  model,  after  the  initial  development,  the  first  version 
enters  the  evolution  stage  where  significant  updates  are  made 
in  an  iterative  fashion.  Even  as  the  current  version  is  being 
supported  through  the  various  stages,  the  development  team 
is  evolving  the  system  to  the  next  major  version  that  will  enter 
its  own  set  of  stages  upon  release.  For  an  example,  Rajlich  and 
Bennett  [8]  pointed  out  that  the  Microsoft  Corporation  uses  this 
model  for  the  production,  evolution,  and  support  of  its  operating 
systems  (e.g.  Windows  XP,  Vista,  7,  etc.). 

Examining  this  model  from  a  systems  perspective,  a  few  obser¬ 
vations  can  be  made.  Independent  of  whether  or  not  the  actual 
software  exhibits  emergent  behavior  in  its  operation,  the  fact  that 
the  system  is  being  evolved  both  within  each  version  and  to  the 
next  version  indicates  that  there  is  a  complex  system  involved.  The 
system  where  the  evolution  is  occurring  is  at  least  one  hierarchal 
level  up  from  the  software  system  and  includes  human  agents  (i.e. 
stakeholders  such  as  developers,  maintainers,  users,  etc.)  interact¬ 
ing  with  each  other.  Also,  the  environment  outside  of  the  open 
system  may  be  changing  (e.g.  competition  with  rival  organizations, 
advancements  in  technology,  etc.),  thus  causing  the  system  to 
adapt  and  evolve. 

Positive  Emergent  Behavior 

Moving  back  down  to  the  software  system  level,  positive 
emergent  behavior  represents  great  potential.  Ideally,  a  devel¬ 
oper  can  design  a  system  so  that  desired  properties  emerge 
while  undesired  behaviors  can  be  suppressed.  This  is  a  difficult 
task,  as  emergent  behavior  is  unpredictable  by  nature.  In  one 
research  paper,  Maciaszek  [10]  prescribed  a  meta-architecture 
for  complex  software  systems  that  was  known  to  produce  the 
desirable  emergent  property  of  adaptability  while  preventing 
other  properties  from  emerging.  This  strategy  can  be  applied  to 
using  design  patterns  to  repeat  positive  results  from  previous 
proven  systems. 

In  other  research,  Olaru,  Gratie,  and  Florea  [1  1]  developed  a 
data  distribution  scheme  using  a  cognitive  Multi-agent  System 
(MAS).  The  overall  concept  is  that  simple  cognitive  agents  that 
have  basic  goals  and  behaviors  are  connected  in  a  network  or 
matrix  configuration.  Data  can  be  introduced  into  the  system 
through  any  of  the  agents.  After  the  data  is  introduced  into  the 
MAS,  it  is  propagated  throughout  the  system  so  that  it  is  avail¬ 
able  to  be  read  from  any  agent  in  the  system.  Thus,  individual 
cognitive  agents  interacting  on  the  local  level  produce  the  emer¬ 
gent  property  of  distributing  the  data  throughout  the  system 
without  any  central  control. 

Negative  Emergent  Behavior 

Even  though  positive  emergent  behavior  in  software  systems 
holds  great  promise  for  the  future,  the  maintenance  manager  or 
developer  of  today  will  more  than  likely  have  to  deal  with  mitigat¬ 
ing  the  undesirable  or  negative  emergent  behaviors  in  complex 
software  systems.  How  does  the  maintainer  troubleshoot  and 
repair  a  problem  that  does  not  originate  in  the  code,  but  instead 
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originates  in  the  interactions  between  agents  in  the  system? 

Mogul  [1  ]  proposed  a  research  agenda  to  come  to  grips  with 
the  negative  emergent  property  (i.e.  misbehavior)  problem  in  the 
software  industry.  Agenda  items  include: 

1 .  Creating  a  taxonomy  of  emergent  misbehavior. 

2.  Creating  a  taxonomy  of  typical  causes. 

3.  Developing  detection  and  diagnosis  techniques. 

4.  Developing  prediction  techniques. 

5.  Developing  amelioration  techniques. 

6.  Developing  testing  techniques. 

Mogul  [1]  provided  preliminary  taxonomies  of  emergent 
misbehavior  and  typical  causes  in  the  paper,  but  indicated  that 
the  other  four  technique  categories  would  be  much  more  chal¬ 
lenging  to  develop  and  implement.  Work  in  these  categories  is 
ongoing  as  is  evident  with  the  following. 

Managing  Negative  Emergent  Behavior  in  Software 

Software  maintenance  is  already  difficult  enough  in  regular 
systems,  let  alone  in  complex  software  systems  with  emergent 
behavior.  Software  project  managers  can  benefit  by  keeping  up 
with  the  software  industry  literature  to  gain  insight  into  potential 
methods  for  mitigating  negative  emergent  behavior. 

Toward  Self-maintaining  Systems 

Pertaining  to  the  software  systems  of  the  near  future,  Gabriel 
and  Goldman  [12]  wrote: 

“Future  innovations  in  software  will  need  to  produce  systems 
that  actively  monitor  their  own  activity  and  their  environment, 
that  continually  perform  self-testing,  that  catch  errors  and 
automatically  recover  from  them,  that  automatically  configure 
themselves  during  installation,  that  participate  in  their  own 
development  and  customization,  and  that  protect  themselves 
from  damage  when  patches  and  updates  are  installed.  Such 
systems  will  be  self-contained,  including  within  themselves 
their  entire  source  code,  code  for  testing,  and  anything  else 
needed  for  their  evolution.” 

To  meet  these  goals,  Gabriel  and  Goldman  [1 2]  proposed  a 
hypothetical  hybrid  autopoietic  and  allopoietic  system.  Accord¬ 
ing  to  Gabriel  and  Goldman  [1 2],  an  autopoietic  system  is  one 
that  is  continually  re-creating  itself  and  allopoesis  is  the  process 
whereby  a  system  produces  something  other  than  the  system 
itself.  In  essence,  the  autopoietic  part  of  the  system  would  con¬ 
centrate  on  keeping  the  system  viable  via  monitoring  the  system 
health  and  taking  corrective  action  if  a  system-threatening 
problem  developed  (e.g.  a  negative  emergent  behavior).  The  al¬ 
lopoietic  part  of  the  system  would  operate  as  programs  do  today 
(i.e.,  perform  the  functions  of  the  system). 

Verifying  Complex  Systems  Through  Formal  Methods 

NASA’s  answer  for  dealing  with  undesirable  emergent  behavior 
in  a  complex  system  may  lie  with  verification  through  a  formal  meth¬ 
ods  cocktail.  Rouff,  Hinchey,  Truszkowski,  and  Rash  [13]  reported 
on  research  into  the  viability  of  utilizing  formal  methods  to  verify  the 
emergent  behavior  of  the  Autonomous  Nano-Technology  Swarm 
(ANTS)  mission  that  may  be  used  to  explore  the  asteroid  belt. 


Basically,  the  mission  entails  1,000  two-pound  autonomous 
space  vehicles  that  will  be  transported  to  the  edge  of  the  as¬ 
teroid  belt.  From  there  the  ANTS  will  self-organize  into  explora¬ 
tion  teams  with  leaders.  Various  instruments  will  be  used  to 
collect  data  from  asteroids  that  will  be  periodically  transmitted 
back  to  earth.  For  autonomous  operation,  the  ANTS  will  need 
to  exhibit  the  properties  of  self-configuration,  self-optimization, 
self-healing,  and  self-protection.  Because  the  ANTS  mission 
will  potentially  depend  on  certain  positive  emergent  behaviors 
to  operate  while  not  developing  any  negative  attributes,  many 
formal  methods  and  techniques  were  considered  for  the  verifica¬ 
tion  of  this  intelligent  swarm.  After  the  evaluation,  the  research 
team  settled  on  a  combination  of  four  current  formal  methods 
that  they  plan  to  integrate  into  one  method  that  is  best  suited  to 
verifying  the  behavior  of  intelligent  swarms.  It  is  conceivable  that 
a  similar  combination  of  formal  methods  could  be  used  to  verify 
other  complex  software  systems  to  prevent  negative  properties 
from  emerging  while  grooming  desired  emergent  behaviors. 

Repairing  Emergent  Behaviors 
Through  Runtime  Feedback 

Lewis  and  Whitehead  [14]  have  conceded  that  many  emer¬ 
gent  behaviors  simply  cannot  be  detected  using  testing  or  other 
verification  techniques.  To  combat  this  problem,  they  developed 
a  system  for  detecting  and  repairing  undesirable  emergent 
behavior  at  runtime.  The  main  component  of  the  system  is  a 
runtime  monitor  named  Mayet.  The  program  they  experimented 
with  was  a  variation  of  the  game  Super  Mario  Brothers. 

For  the  system  to  work,  rules  were  input  into  Mayet  so  it 
would  know  what  undesirable  behaviors  to  look  for  (e.g.  the 
character  gets  stuck  in  an  on-screen  object,  jumps  too  high, 
etc.).  Upon  detecting  an  error,  Mayet  sends  a  message  to  the 
game.  After  the  game  receives  the  message,  the  game  repairs 
the  problem  almost  instantaneously.  A  major  catch  is  that  the 
repair  routines  have  to  be  built  into  the  game.  This  seems  prob¬ 
lematic  because  the  developer  has  to  anticipate  the  possible 
repairs  that  may  be  required  while  the  types  of  behavior  that 
are  supposed  to  be  fixed  are  emergent,  thus  difficult  to  predict. 
Regardless,  the  concept  of  repairing  an  emergent  problem  in  a 
software  system  as  it  is  operating  is  a  step  in  the  right  direction. 

Conclusion 

The  knowledge  and  application  of  systems  theory  and  meth¬ 
odologies  has  become  increasingly  important  for  the  manage¬ 
ment  and  maintenance  of  today’s  increasingly  complex  software 
systems.  The  promise  and  the  problem  of  emergent  behavior 
in  complex  software  systems  is  a  double-edged  sword.  Those 
that  chose  to  ignore  the  implications  of  systems  science  and 
emergent  behavior  will  be  relegated  to  a  reactionary  role.  Project 
and  maintenance  managers  who  embrace  the  systems  sci¬ 
ence  viewpoint  will  be  much  better  prepared  to  be  proactive  in 
controlling  their  software  systems.  The  government  and  civilian 
software  engineering  communities  will  need  to  gain  a  deeper 
understanding  of  how  to  capitalize  on  the  synergies  that  positive 
emergent  properties  can  provide  while  reliably  excluding  nega¬ 
tive  emergent  behaviors  from  software  systems. 
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Developing  a  Model 
for  Simplified  Higher 
Level  Sensor  Fusion 

Mike  Engle,  The  George  Washington  University 
Shahram  Sarkani,  The  George  Washington  University 
Thomas  Mazzuchi,  The  George  Washington  University 

Abstract.  Mulitsensor  data  fusion  (MSDF)  has  been  researched  for  decades  yet 
programs  relying  on  it  to  provide  a  situational,  or  threat,  assessment  continue  to 
be  less  than  successful.  In  order  to  alleviate  the  too-much-information,  too-few- 
analysts  issue,  a  better  approach  must  be  determined.  A  survey  of  recent  and 
current  data  fusions  programs  was  conducted  along  with  a  literature  review  on  how 
different  organizations  handle  a  fusion-based  assessment.  Key  points  found  in  this 
study  were  used  to  develop  an  adaption  of  models  that  can  be  used  to  provide  an 
improved  assessment  while  simplifying  the  process  needed  to  get  there. 

MSDF  programs  have  long  been  a  goal  of  the  DoD  and  the 
warfighter.  It  promises  to  combine  information  from  multiple 
sensors  in  order  to  determine  what  traditionally  could  not  be 
determined  by  one  sensor  alone  either  because  of  technological 
limitations  or  geographic  restrictions.  These  multisensor  systems 
can  be  used  to  increase  geolocation  accuracy,  reduce  uncertainty, 
automatically  extract  man  made  features,  and  quickly  identify 
potential  targets.  When  expanded  to  include  higher-level  fusion 
capabilities,  MSDF  tools  can  help  anticipate  future  actions  of 
these  potential  targets  or  provide  recommendations  for  antici¬ 
pated  decision  points.  It  is  no  longer  enough  to  simply  provide 
image  registration  or  to  combine  sensor  level  information  when 
higher-level  fusion  based  software  promises  improved  situational 
awareness  and  autonomous  decision-making  aids. 

The  demand  for  MSDF  systems  has  only  increased  in  the 
era  of  near  ubiquitous  sensors.  With  more  sensors,  especially 
with  the  move  into  persistence,  come  more  data  and  the  need 
for  more  analysts  to  review  the  data.  The  problem  has  long 
since  arrived  that  there  is  too  much  data  for  too  few  analysts. 
The  goal  is  not  to  replace  the  analyst  but  to  better  enable  them 
to  use  the  information  that  is  already  available.  How  often  has 
the  world  been  surprised  by  a  significant  incident  only  later  to 
find  that  there  were  indicators  available  to  prevent  it?  Events 
like  the  2009  Christmas  Day  Bombing  or  the  Ft  Hood  shooting 
were  preceded  by  sufficient  indicators;  all  that  was  needed  was 
someone  to  piece  together  the  parts  in  a  timely  manner. 

It  should  already  be  clear  why  data  fusion  has  been  researched 
for  decades.  Still  today,  there  are  dozens  of  contractors  and  uni¬ 
versities  dealing  with  multiple  agencies  who  continue  searching 
for  solutions  [1 , 2].  The  National  Geospatial-Intelligence  Agency 
(NGA)  outreach  lists  multi-source  and  multi-l NT  fusion  as  priority 
for  research  and  has  asked  for  help  in  tackling  what  they  con¬ 
sider  a  hard  problem  [3].  In  fact,  the  NGA  has  increased  research 
into  different  fusion  technologies  to  such  an  extent  that  other 
agencies  have  reduced  their  funding  [4]. 
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Unfortunately,  many  of  these  fusion  programs  have  been  less 
than  successful  and  the  golden  age  of  sensor  fusion  has  not  yet 
arrived  [3,  4].  Several  factors  can  be  attributed  to  this  issue.  At 
the  sensor  level,  these  systems  must  combine  data  with  vary¬ 
ing  temporal,  spatial,  spectral  and  radiometric  characteristics. 
They,  “may  be  heterogeneous,  possibly  asynchronous,  and  not 
identically  georeferenced  due  to  motion,  limited  fields  of  view,  or 
constraints  on  power  and/or  the  GPS  signal  [4].”  At  the  program 
level,  problems  have  arrived  from  too  grand  a  goal  to  start  with, 
the  requirements  of  a  wide  range  of  disciplines  not  traditionally 
used  in  systems  or  software  engineering,  and  the  use  of  what 
are  traditionally  very  stove-piped,  isolated  tradecraft. 

Sensor  Fusion  Defined 

There  may  be  as  many  interpretations  about  what  defines  data 
fusion  as  there  are  people  who  are  trying  to  solve  it.  The  sensor 
fusion  domain  not  only  includes  combining  the  outputs  of  single- 
modal,  single-phenomenology  sensors  but  also  the  predictive 
assessments  provided  by  systems  relying  on  multi-platform  (dif¬ 
ferent  unmanned  aerial  vehicles  for  example),  mult-INT  (combin¬ 
ing  multiple  intelligence  types  such  as  imagery  intelligence  and 
electronic  signals  derived  data.  An  instructive  way  to  define  DF 
while  conveying  its  wide  scope  is  to  use  a  process  model.  The 
most  referenced  model  within  the  DoD  appears  to  be  the  Joint 
Director  of  Labs  (JDL)  data  fusion  model  shown  in  Figure  1  [5]. 

The  JDL,  is  an  organization  which  no  longer  exists  but  in 
the  1 980s  they  were  tasked  to  develop  a  model  for  data  fu¬ 
sion.  This  JDL  model,  revised  in  1 999,  was  created  to  show  a 
general  process  of  data  fusion  with  wide  applicability  for  both 
government  and  academia.  It  standardizes  communications 
between  engineers  but  does  not  dictate  the  actual  steps  of  per¬ 
forming  fusion  nor  which  levels  must  be  used.  The  model  shows 
multiple  potential  data  sources  on  the  left  that  can  be  directed 
to  any  of  a  number  of  processes  within  the  fusion  domain  then 
the  resulting  output  provided  on  the  right.  Table  1  provides  a 
description  of  the  most  common  fusion  levels  [6]. 

As  an  example,  detecting  a  manmade  object  at  a  specific  lo¬ 
cation,  classifying  it  as  a  tank  and  even  identifying  it  specifically 
as  a  T-72  tank  is  all  covered  under  Object  Assessment  (level- 1 ). 
Situation  assessment  (level-2)  can  use  priory  information  to 
indicate  that  this  Soviet-designed  main  battle  tank  is  possibly  a 
friendly  unit  of  the  Iraqi  Army.  The  number  found  and  location 
would  further  indicate  unit  size,  if  not  the  exact  unit,  and  possibly 
the  unit’s  disposition  such  as  movement  to  contact.  The  impact 
assessment  (level-3)  could  use  this  information  then  indicate 
that  the  explosions  detected  by  acoustic  sensors  may  not  have 
been  an  attack  directly  on  coalition  forces;  however  units  should 
be  moved  to  support  the  Iraqi  Army. 

Though  this  model  does  not  indicate  a  process  where  one  level 
must  be  met  before  the  next,  programs  traditionally  start  at  level- 1 
then  determine  what  must  be  accomplished  in  order  to  reach  the 
next  level  on  up.  This  has  led  to  a  large  number  of  programs  which 
have  worked  through  level- 1  processing  while  not  too  many  have 
successfully  developed  level-3  [2].  When  working  through  each  level 
in  this  manner  level-3  becomes  an  increasingly  complex  goal.  This 
complexity  is  further  increased  when  the  need  for  increased  situ¬ 
ational  awareness  necessitates  moving  from  the  fusion  of  different 
single-1  NT  sensors  to  the  fusion  of  different  multi-l  NT  sensors. 
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Figure  1  The  revised  JDL  Data  Fusion  Model  (Flail,  Liggins,  &  Llinas,  2009) 


Level 

Name 

Description 

0 

Subobject/  Signal 
Assessment 

Preconditioning  Data  to  correct  biases,  perform  spatial  and  temporal 
alignment.  Also  can  include  feature  extraction  or  signal  detection. 

1 

Object 

Assessment 

Association  of  data  to  estimate  an  objects  or  entity’s  position,  kinematics,  or 
attributes  (including  identity) 

2 

Situation 

Assessment 

Aggregation  of  objects/events  to  perform  relational  analysis  and  estimation  of 
their  relationships  in  the  context  of  the  operational  environment. 

3 

Impact  /Threat 
Assessment 

Projection  of  the  current  situation  to  perform  event  prediction,  threat  intent 
estimation,  own  force  vulnerability,  and  consequence  analysis. 

4 

Process 

Assessment 

Evaluation  of  the  ongoing  fusion  process  to  provide  user  advisories  and 
adaptive  fusion  control  or  to  request  additional  sensor/source  data  (resource 
management) 

Table  1  JDL  Fusion  Levels  1-4  with  descriptions 


An  Analysis  of  Recent  Projects 

An  existing  initiative  already  offers  a  concise  summery  of 
current  technology-based  programs,  the  National  Technology 
Alliance  (NTA).  One  of  the  benefits  provided  by  the  NTA  is 
simplifying  USG  access  to  commercial  technology;  specifically 
dual-use  technology  where  cost-sharing  can  be  attained.  It 
also  provides  an  independent  assessment  and  evaluation  of 
government  users’  needs  and  identifies  optimum  technology 
solutions  to  technical  challenges  [1].  Several  of  these  analyses 
have  covered  data  fusion  research  but  in  2009  the  multi¬ 
source  and  multi-l NT  Fusion  Technology  Survey  and  Analysis 
report  conducted  in  conjunction  with  the  Pennsylvania  State 
University  directly  aligns  with  the  type  of  work  needed.  Though 
this  report  covered  several  hundred  government  and  COTS 
sensor  fusion  solutions,  24  separate  projects  ranging  from 
basic  research  to  tool  development  were  picked  for  additional 
study.  These  projects  represent  the  work  funded  by  a  single 
R&D  office  whose  goal  was  the  advancement  of  available  sen¬ 
sor  fusion  based  tools. 

First,  information  was  collected  from  their  project  summa¬ 
ries  as  a  starting  point  to  show  the  breakdown  of  what  was 
included  in  this  sample  space.  Then  a  more  in-depth  analysis 
into  each  project  was  made  in  order  to  provide  an  indepen¬ 
dent  look  while  ensuring  each  was  evaluated  by  a  single 
person.  This  was  done  to  remove  any  bias  or  at  least  provide  a 
consistent  bias  across  all  24  projects.  Finally,  a  third  look  was 
attempted  after  approximately  one  year  in  order  to  determine  a 
status  update. 


The  assumption  was  that  the  majority  of  the  programs 
would  concentrate  on  both  a  single  intelligence  gathering 
discipline  (INT)  and  lower-level  sensor  fusion  techniques.  Once 
the  information  was  collected,  1 1  of  these  selected  projects 
concentrated  on  a  single  INT  although  most  did  span  across 
multiple  phenomenologies.  Seven  programs  were  described 
as  multi-l  NT,  while  most  of  these  simply  provided  a  common 
geospatial  reference  to  a  specific  type  of  non-geospatial  intel¬ 
ligence  data.  The  remaining  six  included  support  items  such  as 
database  development  and  were  determined  to  not  be  directly 
applicable  to  this  breakdown.  Figure  2  shows  that  of  the  1 8 
represented  projects,  nine  were  considered  level- 1  fusion, 
seven  were  considered  level-2  and  the  final  two  were  consid¬ 
ered  level-3  fusion. 

Represented  Programs  by  JDL  Fusion  Level  - 
1st  Review 


Figure  2  Represented  programs  broken  down  by  JDL 
fusion  level  I  AW  project  description 

The  independent  audit  of  the  1 8  represented  projects 
showed  that  a  total  of  1 5  were  likely  level- 1  fusion  technolo¬ 
gies.  This  left  only  one  of  the  original  seven  level-2  projects 
in  place  to  support  situational  assessment.  The  two  projects 
originally  indicated  as  level-3  fusion  remained  level-3  (Figure  3). 
Of  these  final  two,  one  turned  out  to  be  a  study.  This  study  was 
not  rejected  as  a  level-3  project  because  it  potentially  laid  out 
important  groundwork  for  follow-on  multi-l  NT  work.  However, 
it  did  not  provide  for  any  actual  data  fusion  in  itself.  This  left  a 
single  project  out  of  a  total  of  24  to  possibly  become  a  higher- 
level  data  fusion  based  software  tool. 

During  the  review  several  issues  were  noted.  It  was  found  that 
that  a  large  percentage  of  the  level- 1  fusion  projects  required 
multiple  separate  hard  problems  to  be  answered  in  order  to  be 
successful.  Some  of  these  problems  were  the  same  but  ap¬ 
proached  separately  between  the  separate  projects  and  were 
therefore  redundant  efforts.  In  one  instance  a  problem  was 
worked  though  using  a  supporting  technology  that  was  known 
to  be  untested  and  at  a  very  low  technology  readiness  level. 
Though  there  was  testing  as  part  of  the  normal  tool  develop¬ 
ment  process,  none  of  it  was  meant  to  test  performance  of  in¬ 
dividual  technologies  before  being  integrated  into  the  tool.  This 
shows  that  projects  were  initiated  without  determining  existing 
capability  gaps  and  continued  using  high-risk  methodologies 
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Represented  Programs  by  JDL  Fusion  Level  - 
2nd  Review 


Figure  3  Represented  programs  broken  down  by  JDL  fusion 
levels  determined  by  independent  audit 

After  all  work  was  initially  planned  to  be  complete  a  third  round 
of  review  was  undertaken.  This  review  was  less  successful.  It  was 
not  possible  to  find  the  exact  status  of  any  individual  project  the 
organization  was  working  on.  It  was  only  possible  to  find  artifacts 
of  work  leaving  the  organization.  This  included  projects  being  sent 
out  for  independent  testing  or  transitioning  to  a  semi-operational 
status.  From  what  was  found,  the  initial  24  projects  were  roughly 
correlated  to  only  three  available  MSDF  tools.  These  conclusions 
also  support  previously  cited  literature  stating  that  these  types  of 
programs  tend  to  mostly  be  lower-level  data  fusion  based  with 
few  successful  higher-level  programs. 

Evaluating  Alternative  Approaches  to  Data  Fusion 

After  reviewing  the  types  of  existing  sensor  fusion  programs,  the 
next  step  was  to  evaluate  the  process  other  organizations  used 
to  attain  what  could  be  interpreted  as  level-3  data  fusion.  Areas 
covered  included  legacy  military,  finance,  and  weather  projects.  This 
investigation  converged  on  one  manually  intensive  procedure  that 
closely  parallels  the  MSDF  process  discussed  earlier.  It  is  described 
in  the  Army’s  Field  Manual  on  Intelligence,  FM  2-0  [7]. 

The  Army  defines  a  procedure  through  the  military  deci¬ 
sion  making  process  (MDMP)  to  help  identify  the  most 
important  information  to  a  commander.  This  is  important 
because  it  is  likely  that  there  will  always  be  too  much 


Figure  4  Army  ISR  Task  Development  Process  from  FM  2.0 
which  takes  the  mission,  threat  and  environment  into  account 
to  determine  the  most  significant  intelligence  requirements 


information  available  and  the  commander  does  not  need  to 
track  the  status  and  update  from  each  individual  information 
source.  FM  2-0  includes  this  process  in  the  key  intelligence 
task  “conduct  ISR”  summarized  in  Figure  4.  This  is  an 
exhaustive  and  iterative  procedure  that  involves  several  key 
personnel  with  an  in-depth  understanding  of  the  environ¬ 
ment,  unit  capabilities,  and  what  needs  to  happen  to  affect 
mission  success. 

This  process  starts  with  an  understanding  of  the  mis¬ 
sion  that  needs  accomplished.  Then  different  courses  of 
action  are  developed  which  are  analyzed  against  the  threat 
and  environmental  factors  to  produce  a  set  of  intelligence 
requirements  and  Priority  Intelligence  Requirements  (PIR). 
Information  deemed  sufficiently  important  but  not  necessarily 
mission  impacting  are  Intelligence  Requirements.  Information 
on  hostile  forces  essential  to  support  key  decisions  that  must 
be  made  in  order  to  accomplish  a  mission  is  classified  as 
PIR.  The  process  continues  with  an  analysis  of  all  available 
ISR  (intelligence,  surveillance  and  reconnaissance)  assets 
and  their  capabilities.  This,  along  with  the  initial  MDMP,  helps 
identify  collectable  indicators  of  threat  intentions  and  objec¬ 
tives  which  can  then  be  used  to  task  subordinate  units  and 
ISR  collection  platforms. 

Combining  New  Technology  with  Proven  Process 

A  method  to  simplify  building  a  set  of  fusion  algorithms 
to  take  into  account  any  number  of  sensory  input  and  to  try 
to  think  through  possibly  infinite  scenarios  is  to  start  at  the 
traditional  end  point  (level-3  DF)  to  determine  what  actually 
needs  to  be  assessed  then  move  backwards  by  determin¬ 
ing  what  must  be  obtained  in  order  to  get  what  is  needed.  In 
other  words,  if  the  traditional  progression  of  data  fusion  is 
reversed  and  combined  with  the  Army’s  Intelligence  Synchro¬ 
nization  discussed  in  the  previous  section,  then  a  skeleton 
process  of  simplified  multisensor  data  fusion  starts  to  take 
shape  (Figure  5). 


Figure  5  The  Reverse  Data  Fusion  Model  shown  over  the  initial 
JDL  fusion  model  with  traditional  workflow  indicated 
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Taken  a  step  further,  IRs  can  be  analyzed  using  knowl¬ 
edge  of  the  organization’s  existing  intelligence  capabilities 
to  determine  which  could  be  met  using  an  automated  fusion 
process.  These  would  be  labeled  as  Fusion  Information 
Requirements  (FIRs).  FIRs  are  the  intelligence  requirements 
that  can  be  autonomously  processed  by  current  and  poten¬ 
tial  sensor  and  used  in  fusion  processing.  These  FIRs  are 
broken  down  into  indicators  that  support  the  FIRs  and  can 
be  labeled  as  facts.  These  facts  are  the  actual  observations 
that  can  be  detected  by  any  of  the  available  intelligence 
sensors  and  matched  against  priory  information.  In  other 
words,  these  indicators  are  used  to  support  any  one  of  a 
number  of  situation  assessments  that  have  been  predeter¬ 
mined  as  necessary  in  order  to  match  a  threat  assessment 
or  answer  a  PIR. 

Next,  the  most  likely  methods  to  observe  these  indicators 
are  thought  through.  Each  may  have  multiple  methods  of  de¬ 
tection.  Depending  on  timeliness  requirements,  available  sen¬ 
sors  and  the  environment,  each  reasonable  detection  is  used 
to  create  a  Collection  Requirement  (CR).  CRs  are  the  tasking 
to  the  specific  intelligence  collector  such  as  aircraft,  soldiers  or 
ground  sensors  that  are  most  likely  to  observe  what  is  needed 
in  the  time  frame  that  it  is  needed.  Each  CR  is  added  to  the 
existing  requirements  management  process.  An  example  is 
shown  in  Figure  6  where  three  separate  FIRs  are  broken  down 
into  their  applicable  facts.  FIR-1  needs  three  Facts  meet  in 
order  to  be  satisfied.  Each  fact  can  be  met  through  a  set  of 
detections  using  Boolean  logic. 

Figure  7  shows  how  the  FIRs  (previously  PIRs),  Facts 
(indicators),  and  CRs  loosely  align  with  but  move  opposite  of 
the  more  traditional  object,  situation,  and  threat  assessment 
functions  of  the  JDL  MSDF  model.  This  process  continues  in 
cycles  as  the  threat  evolves,  new  PIRs  are  determined,  or  the 
availability  of  different  ISR  platforms  change.  This  creates  both 
a  synchronized  collection  effort  and  a  modular  approach  to 
MSDF.  It  also  provides  a  basis  for  real-time  information  collec¬ 
tion  and  processing  without  creating  any  redundant  processes 
to  an  organization.  Even  if  the  result  is  only  an  alert  in  an 
operations  center  or  an  email  sent  to  the  responsible  analyst, 
pertinent  and  timely  information  is  sent  to  the  specific  person 
in  need,  in  near  real-time,  without  having  to  monitor  countless 
hours  of  data  feeds. 
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Fact  1  +  Fact  2  +  Fact  3 
Fact  1  -  Detection  1  or  2  &  3 
Fact  2  -  Detection  1  or  4 
Fact  3  -  Detection  5  &  6 

Fact  4  +  Fact  2  +  Fact  5 


Fact  6  +  Fact  7 


Figure  6  PIRs  that  can  be  used  as  fusion  information  requirements 
are  further  broken  down  into  Facts  and  Detections 


Processing/  Assessment 


Figure  7  Reverse  higher  level  data  fusion  model 


Conclusion 

Higher  level  multisensor  data  fusion  programs  allow  for  a 
solution  that  is  more  significant  than  the  sum  of  the  data  sup¬ 
plied  to  them.  In  this  case,  they  take  new  and  known  informa¬ 
tion  and  provide  a  level  of  data  abstraction  in  order  to  help 
understand  what  is  going  on  and  to  do  this  quicker  then  what 
would  normally  be  possible.  This  allows  for  timely  decisions  to 
be  made  as  events  occur  or  statuses  change,  instead  of  after 
analysts  have  had  time  to  analyze  each  situation  manually. 

Care  should  be  taken  to  limit  work  that  is  too  similar  to 
work  already  funded  or  completed.  This  includes  anything 
from  basic  R&D  initiatives  to  acquisition  programs  placing 
major  end  items  into  combat.  Care  should  also  be  taken  to 
limit  the  overall  scope  of  what  a  MSDF  program  may  cover. 

If  the  intent  is  to  develop  a  new  MSDF  system  for  a  specific 
purpose  then  do  not  add  new  and  unrelated  capabilities. 

Many  very  capable  systems  already  exist  but  varying  missions 
and  the  effects  of  rapid  fielding  initiatives  have  limited  their 
capabilities  and  interconnections  into  other  systems.  Using 
principles  of  modular  systems  engineering  and  borrowing 
from  aspects  of  different  levels  of  sensor  fusion  (fusion  at 
the  sensor,  object  or  decision  level)  and  a  simplified  method 
of  improving  the  common  operation  picture  may  be  possible 
while  leveraging  on  existing  capabilities. 

Though  many  MSDF  programs  have  met  with  limited  suc¬ 
cess  it  seems  entirely  possible  that  simply  reversing  the  order 
in  which  most  programs  run  may  affect  positive  outcomes.  By 
taking  what  absolutely  must  be  known  (facts),  finding  ways  to 
first  characterize  then  indicate  these  facts  through  available 
sensor  detections,  then  to  provide  an  output  largely  based 
on  relatively  simple  Boolean  math  and  a  whole  new  model 
for  future  programs  is  created.  When  taken  in  the  traditional 
lower  to  higher  level  DF  order,  advanced  processing  must  be 
developed  in  order  to  account  for  countless  possible  com¬ 
binations  of  unknown  future  indications.  The  reverse  model 
alleviates  the  need  of  this  advanced  methodology,  such  of 
cognitive  engineering  and  neural  networks,  and  simply  waits 
for  detections  that  can  answer  the  commander’s  priority 
information  requirements.^ 
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Basing  Earned 
Value  on  Technical 
Performance 

Paul  Solomon,  PMP 

Abstract.  Previous  articles  in  CrossTalk  and  the  Journal  of  Software  Technol¬ 
ogy  provided  practical  guidance  to  improve  the  quality  of  Earned  Value  Manage¬ 
ment  (EVM)  information  [1 , 2,  3,  4].  This  update  recommends  contract  language 
and  project  monitoring  techniques  to  ensure  that  contractors  integrate  technical 
performance,  including  software  functionality,  with  EVM.  The  key  enablers  are 
the  Integrated  Master  Plan  (IMP)  and  linkage  to  Systems  Engineering  (SE)  work 
products  and  best  practices. 

EVM  can  become  an  effective  program  management  tool  and 
deliver  better  value  to  the  taxpayer  and  warfighter  if  contrac¬ 
tors  revised  their  processes  and  reports  to  integrate  technical 
performance  and  quality  with  cost  and  schedule  performance. 
However,  there  are  no  contractual  requirements  within  the 
acquisition  regulations  or  contract  data  requirements  to  require 
that  contractors: 

1 .  Tie  the  technical  baseline  to  the  EV  Performance 
Measurement  Baseline  (PMB). 

2.  Tie  technical  progress  to  the  Technical  Performance 
Measures  (TPM)  of  the  program,  including  progress 
towards  achieving  planned  functionality. 

EVM  Challenges 

The  guidance  in  this  article  meets  EVM  challenges  that  were 
addressed  in  the  DoD  report  to  Congress;  DoD  EVM:  Perfor¬ 
mance,  Oversight  &  Governance  Report  that  was  required  by  the 
“Weapon  Systems  Acquisition  Reform  Act  of  2009.”  The  chal¬ 
lenges  concern  technical  performance  and  SE,  as  follows. 


SE  Process  and  Products 

EVM  can  be  an  effective  program  management  tool  only  if  the: 

•  EVM  processes  are  augmented  with  a  rigorous  SE  process 

•  SE  products  are  costed  and  included  in  EVM  tracking. 

If  the  SE  lifecycle  management  method  is  integrated  with  the 
planning  of  the  PMB,  then  EVM  will  accurately  measure  techni¬ 
cal  performance  and  progress. 

Contractual  Impediments  to  Effective  EVM 

Neither  the  Defense  Federal  Acquisition  Regulation  Supple¬ 
ment  (DFARS)  nor  the  Data  Item  Descriptions  (DID)  require 
contractors  to  tie  EV  to  technical  performance.  The  DFARS 
Earned  Value  Management  System  (EVMS)  clauses  cite  compli¬ 
ance  with  the  ANSI-748  EVMS  guidelines.  However,  the  use  of 
TPMs  is  optional  per  EVMS.  Per  the  defense  acquisition  pro¬ 
gram  support  methodology,  “EVMS  has  no  provision  to  measure 
quality.  Use  TPMs  to  determine  whether  your  percent  comple¬ 
tion  metrics  accurately  reflect  quantitative  technical  progress 
and  quality  toward  meeting  key  performance  parameters.” 

EVMS  focuses  on  the  work  scope  and  is  silent  on  product 
scope.  It  also  states,  “EV  is  a  direct  measurement  of  the  quantity 
of  work  accomplished.  The  quality  and  technical  content  of  work 
performed  is  controlled  by  other  processes.”  These  loopholes 
create  a  “quality  gap.”  The  quality  gap  enables  contractors 
to  submit  misleading  management  information.  EV  and  the 
cost  performance  may  be  overstated  when  it  is  based  on  the 
percentage  of  drawings  or  code  completed  without  regard  to 
the  technical  maturity  of  the  evolving  design.  As  a  result,  the 
estimate  at  completion  may  be  understated. 

Useful  guidance  to  link  EVM  with  TPMs,  the  technical  base¬ 
lines,  IMP  accomplishment  criteria,  and  SE  work  products  is 
found  in  many  DoD  guides,  as  summarized  at  <http://www.pb- 
ev.com/Pages/DoDGuidance.aspx>.  However,  acquisition  man¬ 
agers  are  not  able  to  implement  this  guidance  if  the  contractors 
fail  to  provide  needed  information.  Even  the  IMP  is  optional  in 
DoD  guidance  and  not  contractually  required  in  DFARS. 


Technical  Performance 

•  EVM  can  be  an  effective  program  management  tool  only  if 
it  is  integrated  with  technical  performance 

•  The  engineering  community  should  establish  TPMs  that 
enable  objective  confirmation  that  tasks  are  complete; 

•  If  good  TPMs  are  not  used,  programs  could  report  1 00% 
of  earned  value  (or  credit  for  work  performed),  even  though 
they  are  behind  schedule  in  terms  of  validating  require 
ments,  completing  the  preliminary  design,  meeting  weight 
targets,  or  delivering  software  releases  that  meet 

the  requirements. 

•  The  EV  completion  criteria  must  be  based  on  technical 
performance,  the  quality  of  work  must  be  verified,  and 
criteria  must  be  defined  clearly  and  unambiguously. 

•  The  PM  should  ensure  that  the  EVM  process  measures 
the  quality  and  technical  maturity  of  technical  work 
products  instead  of  just  the  quantity  of  work  performed. 


Better  Buying  Power:  Suppler  Incentives 

The  DoD  is  striving  to  deliver  better  value  to  the  taxpayer  and 
warfighter  by  improving  the  way  the  it  does  business  via  the 
Better  Buying  Power  (BBP)  initiatives.  BBP  2.0  includes  the 
initiative  to,  “Institute  a  superior  supplier  incentive  program.”  To 
support  that  initiative,  the  Navy  is  currently  developing  a  pilot 
program  for  DoD  with  the  intent  to  recognize  and  reward  con¬ 
tractors  who  demonstrate  superior  performance  by  focusing  on 
cost,  schedule,  performance,  quality,  and  responsiveness. 

The  following  opportunities  and  solutions  should  be  consid¬ 
ered  when  developing  BBP  2.0  supplier  incentives. 

Opportunities  and  Solutions 

The  following  guidance  seizes  four  opportunities  that  underlie 
the  EVM  challenges,  as  shown  in  Figure  1.  Solutions  to  improve 
contractual  requirements  and  acquisition  management  follow. 
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Four  Opportunities 


1.  Base  EV 

on  Technical 
Performance 


Top  Down 


2.  Account  for 

Deferred 

Functionality 


I 


3,  Track  5E 
Tasks 
Discretely 


Planning 

Measure 

Interim 

Progress 


Figure  1 


Base  EV  on  Technical  Performance 

This  opportunity  has  two  components.  First,  do  top  down  plan¬ 
ning  that  includes  defining  milestones  for  achieving  technical 
objectives.  Then  measure  interim  progress  towards  those  meet¬ 
ing  those  objectives. 

Top  Down  Planning 

The  solution  for  basing  EV  on  technical  performance  has  two 
components.  First,  develop  integrated  plans  from  the  top  down, 
starting  with  the  technical  baseline.  Second,  track  progress 
towards  meeting  technical  objectives. 

The  elements  of  effective,  top  down  planning  are: 

1 .  Contractually-required  IMP. 

2.  Use  the  Integrated  Baseline  Review  (IBR)  to  reach 
agreement  on  IMP  accomplishment  criteria  and  to  verify 
that  contractor  integrates  technical  performance  and  SE 
work  products  with  the  Integrated  Master  Schedule  (IMS) 
and  EVM. 

3.  Use  major  technical  reviews  and  EVMS  compliance 
reviews  to  verify  that  contractor  maintains  traceability  from 
IMP  to  IMS  to  Control  Account/Work  Packages. 


First,  make  the  IMP  a  contractual  requirement  with  require- 
ments-based  accomplishment  criteria  that  are  tied  to  the 
technical  baseline.  The  criteria  should  include  the  completion 
of  performance  measures  such  as  Measures  of  Effectiveness 
(MOE),  Measures  of  Performance  (MOP)  and  TPMs  at  key  IMP 
events  such  as  the  System  Functional  Review  (SFR),  Prelimi¬ 
nary  Design  Review  (PDR),  and  Critical  Design  Review  (CDR). 
Examples  of  accomplishment  criteria  are  shown  in  Figure  2. 

Second,  use  the  IBR  to  forge  agreements  and  to  verify  the 
degree  of  integrated  program  management.  Verify  implementa¬ 
tion  of  the  following  during  IBR: 

•  Requirements  traceability  from  the  requirements  data  base 
to  the  IMS  and  from  the  IMS  to  work  package  completion 
criteria. 

•  IMS  includes  interim  and  final  milestones  for  development 
of  SE  work  products  with  criteria  that  are  consistent 
with  the  Contract  Work  Breakdown  Structure  (CWBS). 
The  milestones  include  derived  requirements,  definition 

of  required  functionality  and  quality  attributes,  and 
verification  methods  and  criteria. 

•  Milestones  for  establishing  product  metrics.  MOEs  and 
MOPs  are  defined  at  the  SFR.  TPMs  are  defined  at  the  PDR. 

•  Milestones  with  technical  maturity  success  criteria  including 
TPM  planned  values,  meeting  requirements,  and  percent  of 
designs  complete. 

•  Define  success  criteria  for  event-driven  technical  reviews/ 
IMP  events. 

•  Revise/clarify  criteria  for  CDR  and  subsequent  events 
based  on  knowledge  of  revised  and  derived  requirements 
to  be  met  and  TPM  planned  values. 

•  Flow  down  of  SE  milestones  to  work  packages. 


SFR 

PDR 

CDR 

Functional  Baseline 

Allocated  Baseline 

Product  Baseline 

Accomplishment  Criteria 

1 .  Completed  definition  of  the  required 
system  functionality 

•  Functional  and  interface 
characteristics  of  overall  system 

•  Verification  required  to  demonstrate 
their  achievement 

includes 

•  Detailed  functional  performance 
specification  for  the  overall  system 

•  Tests  necessary  to  verify  and 
validate  system  performance. 

2.  Completed  definition  of  MOEs  and 

MOPs 

3.  All  definitions  above  statused  as 
complete  in  Requirements  Data  Base. 

1 .  Completed  definition  of  the  configuration 
items  (Cl)  making  up  a  system 

•  All  functional  and  interface 

characteristics  allocated  from  the 
top  level  system  or  higher-level  CIs 

•  Derived  requirements 

•  Performance  of  each  lower  level  Cl  in 

the  allocated  baseline 

•  Tests  necessary  to  verify  and  validate 

Cl  performance. 

A  technical  performance  baseline  is  in 
place  down  to  the  subsystem  level,  from 
which  the  system  performance  thresholds 
can  be  compared  and  tracked 

2.  All  TPMs  defined  and  allocated  to 

Interface  Control  Documents  and  sub¬ 
systems). 

3.  All  Key  Performance  Parameters  (KPP), 
MOPs,  and  MOEs  allocated  to  sub¬ 
systems 

4.  All  definitions  above  statused  as 
complete  in  Requirements  Data  Base 

1 .  Completed  definition  of  the  required  system 
functionality 

•  Functional  and  interface  characteristics  of 
overall  system 

•  Verification  required  to  demonstrate  their 
achievement 

includes 

•  Detailed  functional  performance  specification 
for  the  overall  system 

•  Tests  necessary  to  verify  and  validate 
system  performance. 

2.  Completed  definition  of  MOEs  and  MOPs 

3.  All  definitions  above  statused  as  complete  in 
Requirements  Data  Base. 

Measure  Interim 
Performance 

The  solution  to  basing  EV  on 
interim,  technical  performance 
includes  the  following  actions.  First, 
establish  objective  linkage  between 
TPM  planned  values  and  EVM.  For 
physical  objectives,  use  TPMs.  For 
planned  functionality,  base  EV  on 
achieved  functional  requirements. 

Next,  compare  the  EV  schedule 
variance  (converted  to  duration)  with 
the  technical  performance  schedule 
variance.  If  the  variances  are  inconsis¬ 
tent,  perform  a  root  cause  analysis  to 
determine  reasons  for  the  inconsis¬ 
tency.  Then  revise  EV  to  be  consistent 
with  technical  performance. 

If  technical  performance  is  behind 
schedule,  perform  variance  analysis 
and  develop  corrective  actions. 

Then,  revise  the  estimate  to  com- 


Figure  2.  Specified  IMP  Reviews,  Baselines,  Accomplishments/Criteria  P^e  forward  for  work  packages 

with  corrective  actions. 
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Finally,  correct  EV  to  reflect  the  technical  performance  status. 

A  backwards  adjustment  to  EV  is  appropriate  for  work  packages 
with  corrective  actions.  This  technique  enables  the  use  of  EV  to 
track  corrective  actions  to  resolution  and  closure. 

Account  for  Deferred  Functionality 

In  practice,  contractors  seldom  account  for  deferred  functionality 
when  functional  requirements  are  deferred  from  one  build,  release, 
or  block  to  another.  Normally,  the  numbered  build  and  its  respective 
work  package  are  “closed”  and  1 00%  of  the  EV  is  taken,  based  on 
being  finished  with  the  build.  When  this  happens,  EV  fails  to  disclose 
the  true  schedule  variance.  Also,  cost  performance  is  overstated. 

The  solution  is  to  account  for  deferred  functionality.  If  the 
build  is  released  short  of  its  planned  functionality,  the  preferred 
technique  is  to  take  partial  EV  and  close  the  work  package. 
Then,  transfer  the  deferred  scope  and  Budgeted  Cost  of  Work 
Remaining  (BCWR)  to  the  first  month  of  the  work  package  of 
the  next  increment.  When  this  is  done,  EV  mirrors  technical 
performance  and  the  schedule  variance  is  retained. 

Track  SE  Tasks  Discretely 

SE  tasks  are  sometimes  incorrectly  planned  as  level  of  effort. 
Even  when  SE  is  discretely  planned,  EV  is  often  based  on  interim 
milestones  of  progress  towards  completing  a  document  such  as  a 
specification.  These  techniques  fail  to  show  objective  progress  to¬ 
wards  completing  requirements-based  SE  tasks  such  as  require¬ 
ments  analysis  and  validation,  definition  of  technical  measures, 
or  completion  of  trade  studies.  Getting  behind  schedule  on  these 
tasks  is  an  early  indicator  that  an  IMP  event,  such  as  SFR,  PDR, 
or  CDR,  will  slip. 

The  solution  for  measuring  SE  tasks  discretely  has  several 
elements.  First,  include  significant  accomplishments  and  ac¬ 
complishment  criteria  for  SE  tasks  and  work  products  in  the 
IMP.  Next,  show  progress  towards  completing  those  SE  work 
products  in  IMS  and  work  packages.  Typical  SE  work  products 
include  the  system  architecture  (functional  and  physical),  inter¬ 
face  controls,  specifications,  trade  studies,  and  test  procedures. 

For  SE  tasks  such  as  defining  and  approving  the  product 
requirements,  including  derived  requirements  and  allocated 
requirements,  develop  a  requirements-based,  time-phased 
budget  that  is  based  on  the  planned  schedule  for  those  require¬ 
ments.  Then  base  EV  on  the  progress  towards  completing  those 
requirements  as  recorded  in  the  requirements  data  base.  Typical 
examples  of  requirements  status  include  defined,  early  validated, 
determined  verification  method,  approved,  allocated,  and  traced 
to  a  test  procedure. 

For  work  packages  that  result  in  SE  work  products  that  are 
technical  measures  (MOEs,  MOPs,  and  TPMs),  base  EV  on 
progress  towards  meeting  the  IMP  criteria  for  their  completion. 

Plan  Rework  and  Track  it  Discretely 

Rework  is  frequently  not  adequately  planned  in  the  PMB  and  IMS. 
The  rework  can  include  rework  of  requirements  analysis,  design,  and 
test  tasks.  Even  if  rework  is  belatedly  budgeted  from  management 
reserve,  it  is  often  measured  as  level  of  effort,  or  if  measured  discretely, 
as  a  percent  of  the  planned  iterations.  Neither  technique  reports  prog¬ 


ress  towards  developing  or  meeting  the  technical  requirements. 

The  solution  for  better  understanding  and  management  of 
rework  begins  the  proposal  and  the  negotiated  contract  value. 
The  program  should  verify  realistic  that  rework  assumptions 
and  estimates  are  included  in  suppliers’  proposals  and  negoti¬ 
ated  values.  The  estimates  should  include  productivity/quality 
measures  such  as  rework  percent  and  defect  density. 

The  program  should  review  the  adequacy  of  budget  and 
schedule  for  rework  in  the  PMB.  Rework  should  be  planned 
in  a  separate  planning  package  from  the  original  task.  When 
converted  to  a  work  package,  it  should  be  measured  discretely 
based  on  technical  maturity  targets. 

Establish  interim  milestones  for  rework  with  associated  TPM 
planned  values  or  quantified  functionality  based  on  meeting 
requirements.  Then  take  interim  EV  based  on  net  achieved 
technical  performance.  Make  a  negative  adjustment  to  EV  when 
necessary  for  accurate  status  reporting. 

If  rework  is  not  in  a  separate  work  package  and  if  EV  had 
been  taken  for  achieving  a  technical  milestone,  correct  EV  and 
the  IMS  when  there  is  subsequent  knowledge  that  the  mile¬ 
stone  completion  criteria  are  now  unmet.  The  milestone  should 
be  re-opened  and  a  negative  adjustment  should  be  made  to  EV. 
Cumulative  EV  must  reflect  net  technical  progress. 

New  Contractual  Requirements 

New  contractual  requirements  should  be  included  in  the 
Statement  of  Work  (SOW)  to  communicate  program  needs. 
Some  of  the  requirements  are  tantamount  to  tailoring  several  of 
the  EVMS  guidelines.  The  primary  objective  is  to  refocus  man¬ 
agement  attention  from  the  work  scope  to  the  product  scope 
and  to  provide  EV  that  truly  reflects  technical  performance. 
Recommendations  for  acquisition  reforms,  including  a  revision 
to  DFARS,  are  in  a  Defense  AT&L  article  [5].  However,  program 
offices  can  accomplish  the  same  objectives  by  implementing  the 
specific  recommendations  that  follow. 

1 .  For  top  down  planning,  make  the  IMP  a  contractual 
requirement  and  use  a  tailored  CWBS  DID. 

2.  Use  tailored  EVMS  guidelines  or  specify  EVM  techniques 
in  the  SOW  to: 

a.  Incorporate  the  product  scope  or  technical  baseline 
in  the  PMB. 

b.  Tie  EV  to  technical  performance. 

c.  Account  for  deferred  functionality. 

d.  Track  specified  SE  tasks  discretely. 

e.  Plan  rework  and  track  it  discretely. 

IMP  and  SE  Work  Products 

Require  that  an  IMP  be  a  contract  deliverable.  Start  with  the 
DoD  IMP  and  IMS  Preparation  and  Use  Guide  that  is  tailored  to 
specify  SE  work  products  and  accomplishment  criteria.  The  IMP 
DID  should  be  developed  by  the  program  SE  organization. 

An  excellent  source  for  specifying  the  SE  tasks  and  work  prod¬ 
ucts  is  the  Air  Force  Space  and  Missile  Command  Standard,  SE 
Requirements  and  Products  [6].  For  example,  it  states  that  required 
SE  products  are:  the  SE  accomplishments,  accomplishment  criteria, 
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and  narrative  in  the  IMP;  tasks  in  the  IMS;  and  work  packages  in 
the  EVMS,  and  such  other  specific  plans  (such  as  tradeoff  plans) 
as  may  be  needed  to  achieve  the  attributes  required  above. 

CWBS  DID,  DI-MGMT-81334C 

In  practice,  the  CWBS  does  not  include  or  point  to  the  quanti¬ 
fied  technical  or  functional  performance  requirements  that  are  in 
the  specifications.  Contractors  will  have  to  reference  the  functions 
at  the  Cl  level  in  the  allocated  functional  baseline  and  product 
specifications  in  the  product  baseline.  The  contractual  language  is: 
The  CWBS  Dictionary  for  the  appropriate  CWBS  elements  must  be 
updated  to  include  or  reference,  at  PDR,  the  functions  allocated  to 
one  or  more  system  CIs  and,  at  CDR,  the  product  specifications  for 
each  Cl  in  the  system. 

Product  Scope 

With  regard  to  Guideline  2.1a,  authorized  work,  add  contrac¬ 
tual  language  to:  “Include  the  work  necessary  to  produce  the 
product  scope  of  the  program,  including  rework  (when  applica¬ 
ble).  The  product  scope  is  the  technical  baseline.  It  includes  the 
features  and  functions  that  characterize  a  product  or  result.” 

Technical  Performance 

With  regard  to  Guideline  2.2b,  measure  performance,  add 
contractual  language  to  specify  that  “All  TPMs  that  have  been 
identified  at  PDR  shall  be  used  to  measure  progress  in  ap¬ 
propriate  work  packages.  Compare  product  and  process  metrics 
data  against  plans  and  schedule  using  trend  analysis  to  deter¬ 
mine  technical  areas  requiring  management  attention.” 

Deferred  Functionality 

With  regard  to  Guideline  2.5b,  revisions,  add  contractual  lan¬ 
guage  to  specify,  “When  work  scope  that  is  behind  schedule  is 
internally  re-planned  from  the  work  package  that  is  being  closed  to 
another  open  work  package,  the  BCWR  in  the  work  package  that 
is  being  closed  shall  be  transferred  to  the  first  open  period  of  the 
receiving  work  package  The  objective  is  to  prevent  arbitrary  elimi¬ 
nation  of  existing  schedule  variances.  The  time-phased  estimate 
to  complete  of  the  receiving  work  package  must  be  based  on  an 
analysis  of  remaining  tasks  in  the  IMS  and  projected  resource  plan.” 

Rework 

With  regard  to  Guideline  2.1  a,  authorized  work,  add  contrac¬ 
tual  language  to  specify  that:  “The  work  scope  includes  rework. 
Rework  includes  corrective  actions  to  hardware/software  defi¬ 
ciencies,  including  deficiencies  in  the  underlying  requirements. 
Rework  shall  be  planned,  estimated,  and  included  in  the  initial 
PMB.  Rework  shall  be  measured  discretely  and  use  technical 
performance  goals  to  measure  progress. 

Conclusion 

DoD  has  identified  challenges  to  improve  the  usefulness  and 
validity  of  EV  information  by  integrating  technical  performance 
and  systems  engineering  work  products  with  EVM. 


Implementation  of  the  recommended  acquisition  management 
processes  and  new  contractual  requirements  will  provide  the 
following  benefits: 

•  Close  the  EVMS  Quality  Gap 

•  Insightful  IBRs  and  technical  reviews 

•  Valid  contract  performance  reports 

-  Objective  technical/schedule  status 

-  Credible  EAC 

•  Early  detection  of  problems 

-  Program  performance 

-  EV  measurement  and  compliance 

Incentives  for  suppliers  to  implement  these  process  improve¬ 
ments  can  be  implemented  through  the  BBP  2.0  initiatives. 
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Core  Estimating 
Concepts 

William  Roetzheim,  Level  4  Ventures,  Inc. 

Abstract.  Understanding  the  core  estimating  concepts  will  help  you  understand 
any  of  the  currently  available  estimating  tools  and  provide  you  with  the  framework 
you  need  when  building  new  models  for  your  particular  problem  domains.  This  ar¬ 
ticle  strips  off  the  domain  specific  layers  to  get  at  the  basic  skeleton  that  underlies 
estimation  in  general. 


Estimating  Concepts 

Most  estimating  articles,  and  tools,  focus  on  domain  specific 
models,  benchmark  data,  and  approaches.  But  for  all  labor 
related  activities  there  are  some  generic  concepts  that  underlie 
estimates  for  any  of  the  types  of  work  to  be  performed.  These 
fundamental  concepts  apply  whether  you  are  using  commercial 
parametric  estimating  tools  or  home-built  Excel  based  models. 
User  configurable  cost  estimating  tools  can  be  configured  using 
these  core  concepts  to  support  estimates  for  any  labor  driven 
work,  or  even  for  projects  consisting  of  fundamentally  different 
types  of  activities,  even  if  the  tool  originally  ships  pre-initialized 


Figure  1 :  Core  Estimating  Concept,  provides  an  overview  of 
the  estimating  process  at  a  sufficiently  high  level  to  ensure  that 
it  applies  to  estimating  within  any  labor  driven  problem  domain. 

Step  one  in  the  process  is  to  identify  one  or  more  High-level 
Objects  (HLOs)  that  have  a  direct  correlation  with  effort.  The 
HLOs  that  are  appropriate  are  domain  specific,  although  there 
is  sometimes  an  overlap.  Examples  of  HLOs  include  yards  of 
carpet  to  lay,  reports  to  create,  help  desk  calls  to  field,  or  claims 
to  process.  In  activity-based  costing,  these  would  be  the  cost 
drivers.  HLOs  are  often  assigned  a  value  based  on  their  relative 
implementation  difficulty,  thereby  allowing  them  to  be  totaled 
into  a  single  numeric  value.  An  example  is  function  points,  which 
are  a  total  of  the  values  for  the  function  point  HLOs. 

HLOs  may  have  an  assigned  complexity  or  other  defining 
characteristics  that  cause  an  adjustment  in  effort  (e.g.,  simple 
report  versus  average  report).  It  is  also  typically  necessary  to 
have  a  technique  for  managing  work  that  involves  new  develop¬ 
ment,  modifications  or  extensions  of  existing  components,  or 


testing/validation  only  of  existing  components.  Various  formulas 
or  simplifying  assumptions  may  be  used  for  this  purpose.  For  ex¬ 
ample,  in  the  case  of  reuse  the  original  Constructive  Cost  Model 
(COCOMO)  I  model  reduced  the  HLO  size  to: 

HLO  =  HLO  *  (.4DM  +  .3CM  +  .3IT) 


Where  DM  is  the  percent  design  modification  (1%  to  100%); 
CM  is  the  percent  code  modification  (1  %  to  1 00%);  and  IT  is 
the  percent  integration  and  test  effort  (1  %  to  1 00%). 

Step  two  is  to  define  adjusting  variables  that  impact  either 
on  productivity,  or  on  economies  (or  diseconomies)  of  scale. 
The  productivity  variables  tend  to  be  things  like  the  character¬ 
istics  of  the  labor  who  will  be  performing  the  work  or  the  tools 
they  will  be  working  with;  characteristics  of  the  products  to  be 
created  (e.g.,  quality  tolerance)  or  the  project  used  to  create 
them;  and  characteristics  of  the  environment  in  which  the  work 
will  be  performed.  The  variables  that  impact  on  economies 
or  diseconomies  of  scale  are  typically  things  that  drive  the 
necessity  for  communication/coordination,  and  the  efficiency 
of  those  activities.  These  adjusting  variables  are  important  both 
to  improve  the  accuracy  of  any  given  estimate,  and  also  to 
normalize  data  to  support  benchmarking  across  companies  or 
between  application  areas. 

Step  three  involves  defining  productivity  curves.  These  are 
curves  that  allow  a  conversion  between  adjusted  HLO  sizing 
counts  and  resultant  effort.  They  are  typically  curves  (versus 
lines)  because  of  the  economies  or  diseconomies  of  scale  that 
are  present.  Curves  may  be  determined  empirically  or  approxi¬ 
mated  using  industry  standard  data  for  similar  domains.  Curves 
may  also  be  adjusted  based  on  the  degree  to  which  the  project 
is  rushed.  In  any  event,  procedures  are  put  in  place  to  collect  the 
necessary  data  to  support  periodic  adjustment  of  the  curves  to 
match  observed  results,  a  process  called  calibration. 

The  outputs  of  the  process  are  driven  by  the  needs  of  the 
organization.  These  outputs  can  be  broken  down  into  three 
major  categories: 

1.  Cost  (or  effort,  which  is  equivalent  for  this  purpose): 

In  addition  to  the  obvious  total  value,  most  organizations  are 
interested  in  some  form  of  breakdown.  Typical  breakdowns 
include  breakdowns  by  organizational  unit  for  budgetary  or  re¬ 
source  planning  purposes;  breakdowns  by  type  of  money  from  a 
generally  accepted  accounting  principles  perspective  (e.g.,  opex 
versus  capex);  or  breakdown  by  work  breakdown  structure  ele¬ 
ments  in  a  project  plan.  These  outputs  will  also  typically  include 
labor  needed  over  time,  broken  down  by  labor  category.  These 
outputs  are  generated  using  atop  down  allocation. 

2.  Non-cost  Outputs:  Non-cost  outputs  are  quantitative 
predictions  of  either  intermediate  work  product  size,  or  non¬ 
cost  deliverable  components.  Examples  include  the  number 
of  test  cases  (perhaps  broken  down  by  type),  the  engineering 
documents  created  with  page  counts,  the  number  of  use-case 
scenarios  to  be  created,  or  the  estimated  help  desk  calls  broken 
down  by  category.  These  outputs  are  typically  created  using 
curves  similar  to  the  productivity  curves,  operating  either  on  the 
HLOs  or  on  the  total  project  effort. 
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3.  Lifecycle  Costs:  If  the  estimate  is  for  a  product  to  be  cre¬ 
ated,  delivered,  and  accepted  then  the  cost  and  non-cost  items 
above  would  typically  cover  the  period  through  acceptance.  In 
most  cases  there  would  then  be  an  on-going  cost  to  support 
and  maintain  the  delivered  product  throughout  its  lifecycle. 

These  support  costs  are  relatively  predictable  both  in  terms 
of  the  support  activities  that  are  required  and  the  curves  that 
define  the  effort  involved.  For  many  of  them,  the  effort  will  be 
high  immediately  following  acceptance,  drop  off  over  the  course 
of  one  to  three  years  to  a  low  plateau,  then  climb  again  as  the 
product  nears  the  end  of  its  design  life. 

Understanding  these  basic  concepts,  it  is  clear  that  for  a 
given  system  there  may  be  many  different  estimates  that  need 
to  be  prepared  and  combined.  Each  aspect  of  the  work  that 
involves  different  HLOs,  different  adjusting  variables,  or  differ¬ 
ent  productivity  curves  is  really  a  different  model.  But  all  of  the 
models  rest  within  a  consistent  framework,  and  in  fact,  can  run 
within  the  same  tool.  There  is  another  dimension  of  the  estimate 
we  need  to  consider:  project  lifecycle,  or  time. 

For  most  projects,  it  is  impossible  to  completely  and  accu¬ 
rately  define  the  end  product  to  be  delivered.  In  fact,  I  would 
argue  that  the  only  way  to  completely  avoid  uncertainty  in  the 
end  product  is  to  have  an  exact  model  of  the  desired  results  be¬ 
fore  you  start,  and  it  is  unusual  to  have  such  a  model  available. 

In  fact,  most  of  the  effort  spent  on  projects  is  on  a  progressive 
elaboration  of  the  baseline  description  of  what  is  to  be  ultimate¬ 
ly  delivered.  As  shown  in  Figure  2:  Progressively  Elaborated 
Baseline,”  the  baseline  of  what  will  ultimately  be  delivered  is 
progressively  elaborated  throughout  the  life  of  the  project.  Using 
software  as  an  example,  the  requirement  specification  elabo¬ 
rates  the  functional  baseline;  the  design  elaborates  the  require¬ 
ment  specification;  and  the  code  elaborates  the  design. 

As  a  project  moves  through  this  process  of  progressive  elabo¬ 
ration,  the  estimation  models  also  progress  forward  (see  Figure  3: 
Estimating  Lifecycle).  At  the  most  obvious  level,  as  you  under¬ 
stand  the  problem  more  you  can  more  accurately  decompose  the 
work  to  be  performed  and  prepare  an  estimate.  However,  there 
is  another  phenomenon  at  work.  The  actual  estimation  model 
components  will  change  as  you  move  through  the  process.  For 
example,  the  HLOs  that  are  used  to  define  the  product(s)  will 
change,  becoming  more  and  more  granular  as  you  move  forward. 
At  the  high-level  estimate  stage  you  might  think  in  terms  of  a  new 
screen  including  supporting  back-end  processing  and  middleware 
communication  components;  at  the  scope  estimate  you  might 
be  looking  at  a  screen,  a  table,  and  a  new  service;  and  at  the 
validation  estimate  stage  you  might  be  talking  in  terms  of  stored 
procedures  to  be  written.  They  are  all  different  perspectives  of  the 
same  functionality  that  will  ultimately  be  delivered,  but  with  differ¬ 
ent  levels  of  granularity.  However,  the  core  components  of  Figure 
1  are  the  same  for  all  of  these  estimates. 

Not  only  are  better  estimates  possible  as  you  move  through 
the  project  life,  but  the  primary  reason  for  doing  the  estimate  will 
change  over  time.  Take  a  look  at  Figure  4:  Estimating  Purposes. 
Early  lifecycle  high-level  estimates  are  often  used  for  demand 
management.  Projects  are  examined  for  feasibility  and  selected 
based  on  ROI  or  other  financial  measures  that  require  estimates 
to  perform  the  calculations.  Scarce  resources  are  allocated  to 
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Figure  3:  Estimating  Lifecycle 

support  planned  projects  based  on  these  demand  estimates. 
One  characteristic  of  high  level  estimates  is  that  a  significant 
percentage  of  the  projects  that  are  estimated  (as  high  as  90% 
in  some  cases)  are  never  started.  Once  a  project  is  at  least 
partially  funded  and  the  requirements  are  better  understood 
and  defined  (i.e.,  the  baseline  has  been  progressively  elaborated 
one  level),  then  a  scope  level  estimate  can  be  prepared.  In  many 
organizations,  this  is  called  a  “commit”  estimate  because  this  will 
be  the  estimate  used  as  a  basis  for  measuring  project  success 
going  forward.  The  scope  level  estimate  defines  the  project 
baseline  estimate.  Changes  in  scope  are  then  estimated  and, 
if  approved,  those  estimates  are  used  to  modify  the  baseline. 
When  the  project  is  complete,  an  as-built  sizing  is  performed  to 
update  the  organization  historical  database  and  for  calibration. 


Figure  4:  Estimating  Purposes 
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One  final  core  concept  of  cost  estimating  is  worth  discussing: 
The  difference  between  an  estimate  and  a  budget.  An  estimate  is 
defined  as  the  most  likely  outcome  of  a  probabilistic  event,  taking 
into  consideration  everything  that  is  currently  known  about  the 
project  However,  the  estimate  does  not  include  risk,  an  important 
component  of  the  project  budget.  As  shown  in  Figure  5:  Estimating 
versus  Budgeting,  the  estimate  defines  a  starting  baseline.  Your 
risk  management  process  (shown  at  the  top  of  the  figure)  will  then 
determine  the  necessary  funds  for  contingency  funds  and  risk 
response  funds.  Risk  response  funds  are  planned  expenditures 
designed  to  reduce  negative  risk  or  enhance  positive  risk  (oppor¬ 


tunities).  Risk  Response  Funds  will  always  be  a  cost  to  the  project. 
Contingency  funds  are  monies  set  aside  to  deal  with  risks  that  are 
known  but  uncertain.  Generally,  these  will  be  a  net  cost  to  a  project, 
although  in  some  situations  where  risk  management  has  identified 
some  significant  positive  risks,  they  may  actually  reduce  the  project 
budget.  Finally,  the  organization  will  normally  want  to  include  a 
management  reserve  to  allow  for  unknown-unknowns,  or  risks  that 
are  not  discovered  until  later  in  the  project  life. 

Putting  it  All  Together 

Let  us  take  a  look  at  how  all  of  this  fits,  starting  with  a  slide 
prepared  by  the  Naval  Center  for  Cost  Analysis  and  presented  by 
Mr.  Bryan  Flynn  at  the  43nd  Annual  DODCAS  [1  ].  As  shown  in 
Figure  6:  DON  Cost  Estimating  Standard,  the  DON  standard  ap¬ 
proach  aligns  well  with  the  approach  just  described.  We  will  look 
at  it  step-by-step,  using  some  examples  to  explain  the  process. 

Step  1:  Establish  Needs  With  Customer 

While  not  directly  addressed  in  this  article,  this  project  initiation 
step  is  actually  the  most  critical,  yet  the  most  often  overlooked. 

I  often  say  that  good  software  cost  analysis  is  90%  stakeholder 
management,  and  1 0%  math.  And  the  key  to  stakeholder  man¬ 
agement  is  understanding  the  needs  of  the  stakeholders. 

Step  2:  Establish  a  Program  Baseline 

Here  we  are  reviewing  the  business  requirements  and  acquisi¬ 
tion  strategy  (perhaps  captured  in  a  cost  analysis  requirements 
description),  identifying  cost  drivers  or  HLOs  of  this  article;  and 
identifying  risk  areas  (the  start  of  risk  analysis).  For  example,  in 


Matter  Experts 

Figure  6:  DON  Cost  Estimating  Standard 
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conducting  an  analysis  of  a  large  DoD  ERP  implementation,  we 
looked  at  the  available  requirement  document  and  determined  that 
the  most  logical  HLOs  would  be  Reports,  Interfaces,  Conversions, 
Enhancements,  and  Workflows.  We  not  only  collected  together  the 
count  of  each,  but  also  assigned  a  complexity  value  to  each  (very 
low,  low,  average,  high,  very  high)  and  differentiated  between  those 
that  were  new  versus  those  that  were  modified.  For  the  modified 
objects,  we  estimated  the  extent  of  the  modification  (low,  medium, 
high).  In  this  case,  we  had  historic  information  that  allowed  us  to  es¬ 
timate  both  the  relative  effort  for  each  type  of  HLO,  plus  the  spread 
between  very  low  to  very  high  complexity  for  each  type  of  HLO. 

Step  3:  Develop  Baseline  Cost  Estimate 

The  methods  and  models  that  are  mentioned  here  are  our 
productivity  curves.  What  we  want  is  models  or  methods  that  will 
allow  us  to  convert  between  HLOs  and  effort.  Or  more  broadly, 
we  might  say  that  we  are  looking  at  cost  curves  to  convert 
between  HLOs  and  cost,  assuming  that  we  can  develop  models 
encompassing  non-labor  cost  driver  equations. 

The  activity  of  normalizing  data  discussed  here  actually 
happens  at  multiple  points  in  the  process.  First,  HLO  types 
are  normalized  relative  to  each  other  through  some  form  of 
relative  weighting  in  terms  of  effort  (or  cost).  Second,  the  cost 
curves  are  normalized  through  project  specific  adjustments, 
our  adjusting  variables. 

The  cost  estimating  relationships  from  the  figure  are  at  the 
heart  of  the  allocation  process  used  to  generate  our  cost  and 
non-cost  related  outputs. 

For  the  ERP  estimate  that  we  are  using  as  our  example,  we 
first  want  to  estimate  the  total  effort.  For  this  we  start  with  a 
suitable  productivity  model  based  on  the  lifecycle  being  used 
and  the  historic  data  set  used  for  the  analysis.  The  resultant 
equation  is  of  the  form: 

Effort  =  a  *  Size*1 * 3 

Where  a  and  p  are  the  constants  of  the  model  and  Size  is  the 
normalized  total  of  the  HLO  values.  We  then  look  at  project  and 
organizational  specific  adjustments  to  a  and  p.  What  we  are  real¬ 
ly  interested  in  here  are  differences  between  this  project/orga¬ 
nization  and  the  historic  projects  that  we  used.  A  couple  of  good 
sources  to  look  for  potential  changes  and  their  likely  impact  on 
the  variables  are  the  COCOMO  II  environmental  variables  and 
the  IFPUG  General  System  Characteristics,  although  those  are 
by  no  means  the  only  valid  sources. 

Step  4:  Conduct  Risk  and  Uncertainty  Analysis 

The  activities  described  here  deal  with  probabilistic  variances 
in  the  cost  estimate  based  on  uncertainty  in  the  estimation 
process  itself.  While  these  are  certainly  one  source  of  risk,  they 
are  not  the  only  source  of  risk.  It  is  probably  more  generically 
correct  to  follow  the  PM-BOK  approach  described  in  this  article, 
in  which  an  allowance  is  added  to  the  estimate  to  allow  for 
risk  mitigation  activities,  risk  contingency  funds  based  on  the 
expected  value  of  the  risk  factors  at  work,  plus  some  form  of 
management  reserve  based  on  the  risk  tolerance  of  the  organi¬ 
zation  and  the  nature  of  the  project. 


Step  5:  Validate  and  Verify  Estimate 

A  key  mistake  many  novice  estimators  make  is  to  bury  their  head 
in  their  spreadsheets  and  end  up  with  results  that  go  against  com¬ 
mon  sense.  In  the  Naval  Aviation  field,  we  would  have  talked  about 
the  necessity  for  a  pilot  to,  “Get  their  head  out  of  the  cockpit.” 

Of  course,  just  because  an  estimate  goes  against  common 
sense  does  not  mean  it  is  wrong.  I  have  seen  many  situations 
where  the  models  were  right  and  common  sense  was  wrong. 

But  it  does  mean  that  you  should  take  another  look  to  make 
sure  you  are  not  making  an  error  of  some  kind. 

And  of  course,  the  validation  of  an  estimate  may  go  beyond  a  gut 
check.  It  is  often  possible  (and  useful)  to  attack  the  problem  using 
two  or  more  different  approaches  and  to  then  see  if  the  results 
converge.  For  example,  you  might  compare  a  parametric  estimate 
with  a  bottom  up  estimate,  or  you  might  prepare  two  estimates 
using  different  HLOs  as  the  sizing  input.  An  estimate  by  analogy  is 
often  a  good  validation  approach.  This  basically  involves  finding  one 
or  more  other  projects  that  is  similar  to  this  project,  adjusting  for 
any  differences,  and  comparing  the  adjusted  historical  values  to  the 
current  estimate.  Another  approach  that  is  sometimes  used  is  to 
compare  the  results  from  two  or  more  commercial  estimating  tools. 

Step  6:  Present  and  Defend  Estimate 

Yes,  of  course  this  is  necessary.  But  what  is  also  necessary 
is  the  step  of  updating  the  estimate  as  additional  information 
becomes  available  throughout  the  life  of  the  project. 

Conclusions 

My  goal  in  writing  this  article  was  to  define  estimating  in 
terms  of  the  fundamental  concepts  that  would  pertain  no-matter 
what  type  of  estimate  you  were  creating  and  no  matter  what 
tool  you  were  employing.  This  understanding  of  the  big  picture 
is  useful  both  in  understanding  how  estimating  models  and  tools 
work,  and  also  in  developing  new  models  or  tools  for  domains 
where  existing  models  do  not  exist. 
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Statistical  Tune-Up  of 
the  Peer  Review  Engine 
to  Reduce  Escapes 

Tom  Lienhard,  Raytheon  Missile  Systems 

Abstract.  Peer  reviews  are  a  cornerstone  to  the  product  development  process. 
They  are  performed  to  discover  defects  early  in  the  lifecycle  when  they  are  less 
costly  to  fix.  The  theory  is  to  detect  the  defects  as  close  to  the  injection  point  as 
possible  reducing  the  cost  and  schedule  impact.  Like  most,  if  not  all  companies, 
peer  reviews  were  performed  and  data  collected  allowing  characterization  of 
those  reviews.  Data  collected  across  the  organization  showed  that  more  than 
30%  of  the  engineering  effort  was  consumed  by  reworking  products  already 
deemed  fit  for  purpose.  That  meant  for  every  three  engineers  a  fourth  was  hired 
just  to  rework  the  defects.  This  was  unacceptable! 


The  major  contributor  to  this  rework  was  defects  that  escaped 
or  “leaked”  from  one  development  phase  to  a  later  phase.  In 
other  words,  the  peer  reviews  were  not  detecting  defects  in  the 
phase  during  which  they  were  injected.  Defect  leakage  is  cal¬ 
culated  as  a  percentage,  by  summing  the  defects  attributable  to 
a  development  phase  that  are  detected  in  later  phases  divided 
by  the  total  number  of  defects  attributable  to  that  phase.  Defect 
leakage  leads  to  cost  and  budget  over-runs  due  to  excessive 


E  ngineering  Effort 


Figure  1 


rework.  For  some  development  phases,  defect  leakage  was  as 
high  as  75%.  By  investigating  the  types  of  defects  that  go  unde¬ 
tected  during  the  various  development  phases,  corrections  can 
be  introduced  into  the  processes  to  help  minimize  defect  leak¬ 
age  and  improve  cost  and  schedule  performance.  An  organiza¬ 
tional  goal  was  then  set  at  no  more  than  20%  defect  leakage. 

To  perform  this  investigation  and  propose  improvements,  a 
suite  of  Six  Sigma  tools  were  used  to  statistically  tune-up  the 
peer  review  process.  These  tools  included  Thought  Process 
map,  Process  Map,  Failure  Mode  and  Effect  Analysis,  Product 
Scorecard,  Statistical  Characterization  of  Data,  and  a  Design 
of  Experiments. 


Having  been  an  engineer  and  process  professional  for  more 
than  20  years,  I  knew  (or  thought  I  knew)  what  influenced  the 
peer  review  process  and  what  needed  to  be  changed  in  the 
process.  But  when  we  began  the  process,  I  kept  an  open  mind 
and  used  Six  Sigma  tools  to  characterize  and  optimize  the  peer 
review  process. 

The  Thought  Process  Map  was  needed  to  scope  the 
project,  keep  the  project  on  track,  identify  barriers,  and  docu¬ 
ment  results.  It  was  useful  to  organize  progress  and  eliminate 
scope-creep. 


Green  -  Update  Red  -  Question  or  expected  result  Blue  -  Answer  or  actual  result  Underlined  -  Barrier 


Figure  2 


The  Process  Map  was  used  to  “walk  the  process”  as  it  is 
implemented— not  as  it  was  defined  in  the  command  media. 
Inputs,  outputs,  and  resources  were  identified.  Resources  were 
categorized  as  critical,  noise,  standard  operating  procedure  and 
controllable.  The  Process  Map  was  extremely  useful  because 
it  quickly  highlighted  duplicate  activities,  where  implementation 
deviated  from  the  documented  process,  and  was  used  as  an  in¬ 
put  to  the  Failure  Mode  and  Effect  Analysis  (FMEA)  and  Design 
of  Experiments  (DOE). 

The  FMEA  leveraged  the  process  steps  from  the  Process 
Map  to  identify  potential  failure  modes  with  each  process  step, 
the  effect  of  the  failure,  the  cause  of  the  failure,  and  any  current 
detection  mechanism.  A  numerical  value  was  placed  on  each  of 
these  attributes  and  a  cumulative  Risk  Priority  Number  (RPN) 
was  assigned  to  each  potential  failure.  The  highest  RPNs  were 
the  potential  failures  that  needed  to  be  mitigated  or  eliminated 
first  and  would  eventually  become  the  factors  for  the  DOE. 

The  Product  Scorecard  contained  all  of  the  quantifiable  data 
relating  to  the  peer  reviews.  It  showed  the  number  of  defects  in¬ 
troduced  and  detected  by  phase,  both  in  raw  numbers,  percent¬ 
age,  and  by  effort.  Using  Pareto  Charts,  it  was  easy  to  determine 
where  defects  entered  the  process,  where  defects  were  found 
by  the  process,  and  even  which  phases  had  the  most  impact 
(rework)  to  the  bottom  line.  Surprisingly,  58%  of  the  total  de¬ 
tects  were  found  in  test,  well  after  the  product  is  deemed  “done”. 
Additionally,  three  phases  accounted  for  greater  than  92%  of 
rework  due  to  defects. 
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Phase  Detected 


—  Test 


Figure  3 

An  improvement  goal  was  set  by  the  organization.  The  immedi¬ 
ate  goal  was  set  around  finding  the  defects  earlier  in  the  lifecycle 
rather  than  trying  to  reduce  the  number  of  defects.  If  the  process 
could  be  improved  to  find  the  defects  just  one  phase  earlier  in  the 
lifecycle,  the  result  would  be  many  hundreds  of  thousands  of  dol¬ 
lars  to  the  bottom  line! 


—  Existing  Detection 
Improved  Detectior| 


Figure  4 


Phase  Detected 

Same  number  of  total  defects  introduced  in  the  same  phases 


The  data  from  the  Product  Scorecard  was  plotted  to  create  a 
distributional  characteristic  of  the  process  capability.  Visually,  this 
highlighted  the  lifecycle  phases  that  were  well  below  our  goal  of 
finding  80%  of  defects  in  phase,  as  seen  in  the  figure  below. 


Going  into  this  project,  my  belief  was  that  a  program  could 
be  identified  that  was  conducting  peer  reviews  effectively 
across  the  entire  lifecycle  and  that  program’s  process  could  be 
replicated  across  the  organization.  The  Control  Chart  showed 
something  quite  different.  All  the  programs  were  conducting 
peer  reviews  consistently,  but  the  variation  between  lifecycle 
phases  ranged  widely.  When  the  data  was  rationally  sub¬ 
grouped  by  phase,  the  data  became  stable  (predictable)  within 
the  subgroups,  but  there  was  extensive  variation  between  the 
subgroups.  This  meant  the  variation  came  from  the  lifecycle 
phases  not  the  programs.  It  would  not  be  as  simple  as  finding 
the  program  that  conducted  effective  peer  reviews  and  replicat¬ 
ing  its  process  across  the  organization. 


P  Chart  for  In  Phase 


P  Chart  for  In  Phase 


Figure  5 


P  Chart  for  In  Phase 


Figure  6 
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The  Analysis  of  Variation  confirmed  that  72%  of  the  process 
variation  was  between  the  subgroups  (lifecycle  phases)  and  only 
28%  was  within  the  subgroup  (programs).  Since  the  data  was 
only  a  sample  of  the  population,  Confidence  Intervals  were  con¬ 
ducted  to  find  out  the  true  range  of  the  population.  This  quickly 
showed  that  for  the  Requirements  Phase,  the  best  the  process 
was  capable  of  achieving  was  detecting  37%  of  defects  in 
phase.  In  fact,  if  no  action  was  taken  it  was  95%  certain  that  the 
Requirements  Phase  will  find  between  21%  -  37%  of  defects 
in  phase,  the  Design  Phase  will  find  between  42%  -  88%  of 
defects  in  phase,  the  Implementation  Phase  will  find  59%  -  78% 
of  defects  in  phase.  This  helped  focus  where  to  concentrate  the 
improvement  resources. 


Analysis  of  Variance  for  percent: 
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Figure  7 


Graphical  analysis  included  a  normal  probability  plot  and  a 
Pareto  chart  of  the  main  effects,  two-way  and  three-way  effects. 
This  clearly  showed  that  training,  criteria,  and  experience  were  the 
influential  factors. 


^1  low  *  Ti  <i  n  lii  .  Ci  ii  ei  i -iii,  pci 
as  tlic  influential  factors 


Remember  the  high  RPNs  from  the  FMEA?  These  were  used 
as  the  factors  in  a  DOE.  There  were  four  factors  (experience, 
training,  review  criteria,  and  number  of  reviewers).  The  response  Figure  9 
variable  for  the  DOE  was  the  percentage  of  defects  found  in  a 
peer  review.  There  were  1 6  runs,  which  made  it  a  half-factorial 


DOE. 

There  were  some  limitations  with  this  DOE.  The  products 
reviewed  were  different  for  each  run;  there  were  restrictions  on 
randomization;  and  by  the  latter  runs  it  was  hard  to  find  a  peer 
review  team  that  fulfilled  the  factor  levels.  For  example,  once 
somebody  was  trained  they  could  not  be  untrained. 

When  analyzing  data,  always  think  golf  (PGA  =  practical, 
graphical,  and  analytical).  Practical  analysis  looked  at  the  result 
of  each  for  anything  of  interest.  It  was  not  until  then  the  runs 


Analytical  analysis  not  only  showed  the  same  influential  factors 
but  also  quantified  the  effect  and  indicated  whether  to  set  the  fac¬ 
tor  high  or  low.  Training  was  the  most  influential,  followed  closely  by 
experience.  The  process  was  relatively  robust  with  respect  to  the 
language  and  number  of  people.  If  peer  reviews  are  just  as  effec¬ 
tive  with  half  the  people,  this  alone  could  have  a  big  savings  to  the 
bottom  line.  The  eye-opener  here  was  that  the  peer  review  process 
was  more  effective  without  criteria.  This  went  against  intuition,  but 
was  based  on  data. 


were  sorted  by  response  did  any  trends  appear.  The  highest  five 
runs  all  had  no  criteria,  the  lowest  four  consisted  of  inexperi¬ 
enced  team  members  and  six  of  the  top  seven  were  ■ 
teams. 
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Further  investigation  revealed  that  the  use  of  criteria  was  re¬ 
stricting  what  the  reviewers  were  looking  for  in  the  peer  reviews. 
Training  was  developed  to  educate  the  reviewers  on  how  to  use  the 
criteria  The  criteria  became  a  living  document  and  as  defects  were 
found  the  checklist  were  updated. 

The  results  showed  remarkable  improvements.  The  number  of 
defects  introduced  before  and  after  improvements  were  in  the 
same  order  of  magnitude  (1947  vs.  1  1 66)  so  that  comparisons 
could  be  made  between  the  “before”  and  “after”  states.  If  you  only 
look  at  the  percentage  of  defects  found  in  phase,  as  a  lot  of  organi¬ 
zations  do,  the  results  can  be  misleading.  It  shows  that  five  of  eight 
phases  actually  found  fewer  defects  in  phase.  Analyzing  the  data 
this  way  assumes  all  defects  are  created  equal  (it  takes  the  same 
amount  of  effort  to  fix  the  defect)  and  does  not  take  into  effect  the 
number  of  phases  the  defect  leaked. 

If  the  defects  are  transformed  into  the  amount  of  rework,  a 
completely  different  profile  is  observed.  In  those  five  phases  that 
found  a  smaller  percent  of  defects  in  phase  the  amount  of  rework 
decreased  by  75%.  Looking  at  the  three  phases  that  accounted  for 
92%  of  the  rework,  the  improvements  are  dramatic.  It  can  be  con¬ 
fidently  stated  that  two  of  the  three  phases  will  exceed  the  goal  of 
finding  80%  of  defects  in  phase.  The  third  phase  only  allowed  1% 
of  the  defects  to  make  it  to  test,  whereas  before  the  improvements, 

1 4%  made  it  to  test.  This  reduced  the  rework  from  1 56  days  to  a 
mere  1 3.  Remember,  measure  what  are  you  trying  to  improve— is  it 
number  of  defects  or  rework? 

The  bottom  line  savings  exceeded  the  goal  by  more  than  20%. 
There  was  a  nominal  increase  in  cost  in  the  early  stage  but,  as 
can  be  seen  by  the  graph,  the  cost  of  rework  leveled  off  after  the 
implementation  phase.  This  means  almost  no  defects  leaked  into 
the  testing  phase  or  beyond.  Imagine  your  organization  having  no 
defects  leak  beyond  the  implementation  phase.  It  can  be  done!^ 

Disclaimer: 

CM  Ml®  and  CMM®  are  registered  in  the  U.S.  Patent  and  Trade¬ 
mark  Office  by  Carnegie  Mellon  University. 


ARDi  IT  THF  AUTHOR 

Tom  Lienhard  is  a  Sr.  Principal  Engineer  at 
Raytheon  Missile  System’s  Tucson  facility 
and  a  Six  Sigma  BlackBelt.  Tom  has  partici¬ 
pated  in  more  than  50  CMM®  and  CM  Ml® 
appraisals  both  in  DoD  and  Commercial 
environments  across  North  America  and 
Europe  and  was  a  member  of  Raytheon’s 
CM  Ml  Expert  Team.  He  has  taught  Six 
Sigma  across  the  globe,  and  helped  various 
organizations  climb  the  CMM  and  CMMI 
maturity  levels,  including  Raytheon  Missile 
System’s  achievement  of  CMMI  Level  5. 

He  has  received  the  AlliedSignal  Quest 
for  Excellence  Award,  the  Raytheon  Tech¬ 
nology  Award  and  the  Raytheon  Excellence 
in  Operations  and  Quality  Award.  Tom  has  a 
BS  in  computer  science  and  has  worked  for 
Hughes,  Raytheon,  AlliedSignal,  Honeywell 
and  as  a  consultant  for  Managed  Process 
Gains. 


Flwi+DnrtnHd 


Figure  13 


CrossTalk-January/Februcry  2013  37 


UPCOMING  EVENTS 


38 


Upcoming  Events 

Visit  <http://www.crosstalkonLine.org/events>  for  an  up-to-date  List  of  events. 


Annual  Computer  Security  Applications  Conference 

3-7  December  201  2 
Orlando,  FL 
http://www.acsac.org 

International  Conference  on  Computing 
and  Information  Technology 

14-15  Jan  2013 
Zurich,  Switzerland 

http://www.waset.org/conferences/201 3/zurich/iccit/ 

Technology  Tools  for  Today  (T3)  Conference 

11- 13  Feb  2013 
Miami,  FL 

http://www.technologytoolsfortoday.com/conference.html 

Strata  Conference:  Making  Data  Work 

26-28  Feb  2013 
Santa  Clara,  CA 

http://strataconf.com/strata201 3 

Software  Assurance  Forum  -  March  2013 

5-7  March  2013 
Gaithersburg,  MD 

https://buildsecurityin.us-cert.gOv/bsi/events/1 41 7-BSI.html 

Government  Contracting 

12- 13  March  2013 
Washington,  DC 

http://publiccontractinginstitute.com/events 

Conference  on  Systems  Engineering  Research 

19-22  March  2013 
Atlanta,  GA 

http://cser1 3. gatech.edu 

7th  International  Symposium  on  Service 
Oriented  System  Engineering 

25-28  March  2013 
San  Francisco,  CA 

http://sei.pku.edu.cn/conference/sose201 3 

Symposium  of  Mobile  Cloud,  Computing 
and  Service  Engineering 

25-28  March  2013 
Redwood,  CA 

http://www.engr.sjsu.edu/gaojerry/ 

IEEEMobileCloud201 3/index.htm 
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Software  Technology  Conference 

8-11  April  2013 

Salt  Lake  City,  UT 

http://www.sstc-online.org 

7th  Annual  IEEE  Systems  Conference 

15-18  April 
Orlando,  FL 
http://ieeesyscon.org 

Systems  Engineering,  Test  and  Evaluation  Conference 

29  April  -  1  May  2013 
Canberra,  Australia 

http://sapmea.asn.au/conventions/sete201 3 

IBM  Edge  2013 

10-14  Jun  2013 

Las  Vegas,  NV 

http://www.ibm.com/edge 

23rd  Annual  INCOSE  International  Symposium 

24-27  Jun  2013 
Philadelphia,  PA 

http://www.incose.org/symp201 3 


BACKTALK 


Writing  Good  Software 

Lessons  Learned  The  Hard  Way 


In  1986,  I  was  one  of  the  founders  of  the  Advanced  Soft¬ 
ware  Engineering  Education  and  Training  (ASEET)  team.  We 
were  composed  of  members  from  all  four  services,  plus  civilian 
contractors.  Our  mission  was  to  spread  the  word  about  software 
engineering  and  how  it  was  much  more  than  just  programming. 
ASEET  is  long  gone,  but  I  still  teach,  present,  and  consult.  After 
years  of  working  with  universities,  military  organizations  and 
DoD  contractors,  I  started  to  learn  (often  the  hard  way)  what  the 
elements  of  good  software  engineering  really  are. 

I  boil  it  down  to  four  attributes-software  must  be  reliable, 
understandable,  modifiable,  and  efficient. 

You  can  easily  add  other  items;  affordable,  usable,  delivered 
on  time,  fault  tolerant,  and  more.  However,  I  have  found  that  the 
above  framework  covers  the  attributes  I  expect  in  “good”  code. 
When  I  teach  software  engineering,  I  use  the  above  list.  I  find  it 
so  important  that  I  put  some  form  of  question  about  this  list  on 
every  test  I  give.  This  list  is  not  my  creation.  I  got  this  framework 
from  Software  Engineering  with  Ada ,  by  Grady  Booch,  back  in 
1 983.  Mr.  Booch  was  Chief  Scientist  for  Rational  Computers, 
one  of  the  developers  of  the  Unified  Modeling  Language,  and 
now  is  with  the  IBM  Thomas  J.  Watson  Research  Center,  serv¬ 
ing  as  Chief  Scientist  for  Software  Engineering.  If  this  list  was 
good  enough  for  him,  it  is  good  enough  for  me.  (As  a  side  note, 
he  also  taught  at  the  USAF  Academy.  I  brag  that  I  was  the  last 
person  to  hold  the  “Booch  Chair  of  Software  Engineering”  when 
I  taught  at  the  Academy.  It  was  not  really  a  formal  position.  One 
day,  somebody  mentioned  to  me  that  I  was  using  Grady  Booch’s 
old  chair  and  office.  The  chair  was  replaced  while  I  was  there— 
so  nobody  else  held  the  chair  after  me.) 

I  want  to  include  1 5  lessons  I  have  learned  over  the  years. 
Some  lessons  were  easy  to  learn;  some  are  ones  I  paid  dearly 
for.  Some  are  my  own;  some  are  lessons  that  I  was  wise  enough 
to  learn  from  others. 

1 .  Always  follow  the  Attributes  of  Good  Software  listed  above. 

2.  You  need  a  coding  standard  and  the  self-discipline  to 
follow  it. 

3.  Document  why,  not  what. 

4.  Code  as  if  the  next  person  to  maintain  your  code  is  a 
homicidal  maniac  who  knows  where  you  live.  (John  F.  Woods) 

5.  A  software  engineering  expert  is  a  person  who  knows 
enough  about  what  is  really  going  on  to  be  scared.  (Adapted 
from  P.  J.  Plauger) 

6.  The  foolish  learn  from  experience.  The  wise  learn  from 
the  experience  of  others. 


7.  Want  to  be  a  good  programmer?  Maintain  your  own 
code  after  letting  it  sit  for  a  few  months. 

8.  Want  to  be  a  really  good  programmer?  Maintain  somebody 
else’s  code  (and  everybody  else’s  code  will  always  be  “bad") 

9.  Want  to  make  your  code  foolproof?  Not  likely.  They  are 
breeding  a  better  quality  of  foolish  users  at  an  amazing  rate. 

1 0.  Need  the  code  really  bad?  We  can  deliver  it  that  way  if 
you  do  not  quit  rushing  us.  (Courtesy  of  many  friends  -thanks 
to  Gene,  Les,  Lindy,  and  all  of  the  other  founding  members  of 
the  ASEET  team.  And  thanks  to  all  of  the  members  during  the 
1 5  years  the  ASEET  existed.) 

1 1 .  You  cannot  have  too  many  backups. 

1  2.  If  you  can  overload  basic  operations  (such  as  “+", 
etc.),  then  your  language  values  “cool"  over  maintain¬ 
ability.  I  do  not  want  to  see  “x  =  x+1 "  and  have  to  debug  for 
hours  to  find  out  that  it  is  not  adding  1 ,  but  instead  invok¬ 
ing  a  user-defined  “+"  function  with  bizarre  side  effects. 
“DoSomeWeirdOperation(x)"  makes  debugging  and  mainte¬ 
nance  a  lot  easier.  (Discovered  independently  by  legions  of 
Ada  and  C++  programmers.  In  one  program  I  was  debugging, 

I  eventually  discovered  “a  <  b"  was  calling  a  user-defined 
function,  but  “a>b"  was  not.  Given  1  2  programmers  for  a  jury,  I 
thought  I  could  get  off  with  “justifiable  homicide") 

13.  In  many  languages  “float_num  =  integer_num1  /  inte- 
ger_num2"  does  integer  division,  truncates,  and  then  converts 
to  a  float,  “y  =  3/2;"  is  always  going  to  be  1 ,  even  if  “y"  is  a 
float.  (This  is  why  I  like  strongly  typed  languages,  which  flag  it  a 
syntax  error.  I  really  learned  this  the  hard  way!) 

14.  Indexing  strings  and  arrays?  If  you  switch  from  languag¬ 

es  that  start  indexing  at  0  to  languages  that  start  at  1  (or  vice 
versa)  you  are  going  to  write  loops  with  an  “off  by  1 "  error.  (Re¬ 
learned  every  time  I  switch  from  Ada  to  C/C++  to  Ada . ) 

1 5.  Always  carry  a  short  extension  cord  when  you  travel. 
When  you  need  to  charge  your  computer,  tablet  and  phone  all 
at  the  same  time,  inevitably  the  single  outlet  in  the  hotel  room 
will  be  behind  the  TV  stand.  (Nothing  to  do  with  software,  but 
truly  a  lesson  I  have  learned  the  hard  way.) 

E-mail  me  your  own  Lessons  Learned  The  Hard  Way.  I  will  try 
and  publish  them  in  a  later  column. 

David  A.  Cook,  Ph.D. 

Stephen  F.  Austin  State  University 
cookda@sfasu.edu 
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N  AV 


★joint  Mission  Planning  System 

★  Battle  Control  System-Fixed 

★  Satellite  Technology 

★  Expeditionary  Fighting  Vehicle 

★  F-16,  F-22,  F-35,  New  Workloads 
Coming  Soon 

★  Ground  Theater 
Air  Control  System 

★  Human  Engineering 
Development 


iS  Like 


★  Health  Care  Packages 

★  10  Paid  Holidays 

★  Paid  Sick  Leave 

★  Exercise  Time 

★  Career  Coaching 

★  Tuition  Assistance 

★  Retirement  Savings  Plans 

★  Leadership  Training 


www.  facebook.  com/309SoftwareMaintenanceGroup 


Email:  309SMXG.SODO@hill.af.mil 
Phone:  (801)  775-5555 


★  25  minutes  from  Salt  Lake  City 

★  Utah  Jazz  Basketball 

★  Three  Minor  League 
Baseball  Teams 

★  One  Hour  from  12  Ski  Resorts 

★  Minutes  from  Hunting,  Fishing, 
Water  Skiing,  ATV  Trails,  Hiking 


Contact  Us: 


Exciting 
and  Stable 
Workloads: 


Employee 

Benefits: 


Location, 

Location, 

Location: 


R 


Crosstalk  thanks  the 
above  organizations  for 
providing  their  support. 


